SlideShare una empresa de Scribd logo
1 de 71
Descargar para leer sin conexión
Friday, January 31, 14
Friday, January 31, 14
Best Practices and
Lessons Learned from
100,000 A/B Tests
Martijn Geerlings

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
9:1
Friday, January 31, 14
Lessons Learned

Friday, January 31, 14
Lesson #1:
Define success .

Friday, January 31, 14
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Friday, January 31, 14
visitors
per day

Friday, January 31, 14

Visit
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
visitors
per day

Friday, January 31, 14

Visit

sign up
rate

Email

$ per recipient

$
Media
Button

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Button: “Sign Up”
Friday, January 31, 14
Button: “Learn More”
Friday, January 31, 14
Button: “Join Us Now”
Friday, January 31, 14
Button: “Sign Up Now”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Media: “Get Involved”
Friday, January 31, 14
Media: “Family”
Friday, January 31, 14
Media: “Change”
Friday, January 31, 14
Media: “Barack’s Video”
Friday, January 31, 14
Media: “Springfield Video”
Friday, January 31, 14
Media: “Sam’s Video”
Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment
Variations:
Button:

Media:

1. Sign Up

1. Get Involved Image

2. Learn More

2. Family Image

3. Join Us Now

3. Change Image

4. Sign Up Now

4. Barack’s Video
5. Springfield Video
6. Sam’s Video

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment

Friday, January 31, 14
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Splash page experiment
Email
Volunteers Amount Raised
Subscriptions
Original:

7,120,000

+40.6%

+2,880,000

+288,000 +$57,000,000

New:

10,000,000

1,000,000 $200,000,000

Friday, January 31, 14

712,000

$143,000,000
Lesson #1:
Define quantifiable success metrics.

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
+43.8%
Purchases

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

+14.8% conversion rate

Friday, January 31, 14
Lesson #2:
Less is more. Reduce choices.

Friday, January 31, 14
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+14.6% clicks

Friday, January 31, 14
Donation button experiment
Variations

Friday, January 31, 14
Donation button experiment
Previously
Donated

0.0%

0.0%

+2.3%

+27.8%

+16.3%

-27.8%

N/A

N/A

+15.2%

-24.6%

+11.9%

+8.5%
Friday, January 31, 14

Signed Up,
Never Donated

0.0%

Variations

Never
Signed Up

+2.9%

+18.4%
Lesson #3:
Words matter. Focus on your call to action.

Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.

-10% conversions
Friday, January 31, 14
Lesson #4:
Fail fast.

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
Friday, January 31, 14
+0.5% Retention
+1.4% Engagement

Friday, January 31, 14
Lesson #5:
Seek the Global Maximum.

Friday, January 31, 14
Lessons Learned
1. Define quantifiable success metrics.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Fail fast.
5. Seek the Global Maximum.

Friday, January 31, 14
Thank You

Friday, January 31, 14
vs.

Friday, January 31, 14
vs.
+8% engagement
Friday, January 31, 14
2009!

Friday, January 31, 14

2010!

2011!

2012!

2013!

Más contenido relacionado

Último

Último (20)

BLOOM_April2024. Balmer Lawrie Online Monthly Bulletin
BLOOM_April2024. Balmer Lawrie Online Monthly BulletinBLOOM_April2024. Balmer Lawrie Online Monthly Bulletin
BLOOM_April2024. Balmer Lawrie Online Monthly Bulletin
 
Digital Strategy Master Class - Andrew Rupert
Digital Strategy Master Class - Andrew RupertDigital Strategy Master Class - Andrew Rupert
Digital Strategy Master Class - Andrew Rupert
 
Unlocking the Mystery of the Voynich Manuscript
Unlocking the Mystery of the Voynich ManuscriptUnlocking the Mystery of the Voynich Manuscript
Unlocking the Mystery of the Voynich Manuscript
 
Defining Marketing for the 21st Century,kotler
Defining Marketing for the 21st Century,kotlerDefining Marketing for the 21st Century,kotler
Defining Marketing for the 21st Century,kotler
 
Brand Strategy Master Class - Juntae DeLane
Brand Strategy Master Class - Juntae DeLaneBrand Strategy Master Class - Juntae DeLane
Brand Strategy Master Class - Juntae DeLane
 
Foundation First - Why Your Website and Content Matters - David Pisarek
Foundation First - Why Your Website and Content Matters - David PisarekFoundation First - Why Your Website and Content Matters - David Pisarek
Foundation First - Why Your Website and Content Matters - David Pisarek
 
Developing Marketing Strategies and Plans kotler
Developing Marketing Strategies and Plans kotlerDeveloping Marketing Strategies and Plans kotler
Developing Marketing Strategies and Plans kotler
 
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 150 Noida Escorts >༒8448380779 Escort Service
 
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO SuccessBrighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
Brighton SEO April 2024 - The Good, the Bad & the Ugly of SEO Success
 
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptx
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptxDigital-Marketing-Into-by-Zoraiz-Ahmad.pptx
Digital-Marketing-Into-by-Zoraiz-Ahmad.pptx
 
Social Media Marketing PPT-Includes Paid media
Social Media Marketing PPT-Includes Paid mediaSocial Media Marketing PPT-Includes Paid media
Social Media Marketing PPT-Includes Paid media
 
Podcast Marketing Master Class - Roger Nairn
Podcast Marketing Master Class - Roger NairnPodcast Marketing Master Class - Roger Nairn
Podcast Marketing Master Class - Roger Nairn
 
Unraveling the Mystery of the Hinterkaifeck Murders.pptx
Unraveling the Mystery of the Hinterkaifeck Murders.pptxUnraveling the Mystery of the Hinterkaifeck Murders.pptx
Unraveling the Mystery of the Hinterkaifeck Murders.pptx
 
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
Generative AI Master Class - Generative AI, Unleash Creative Opportunity - Pe...
 
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort ServiceBDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
BDSM⚡Call Girls in Sector 144 Noida Escorts >༒8448380779 Escort Service
 
Instant Digital Issuance: An Overview With Critical First Touch Best Practices
Instant Digital Issuance: An Overview With Critical First Touch Best PracticesInstant Digital Issuance: An Overview With Critical First Touch Best Practices
Instant Digital Issuance: An Overview With Critical First Touch Best Practices
 
personal branding kit for music business
personal branding kit for music businesspersonal branding kit for music business
personal branding kit for music business
 
Kraft Mac and Cheese campaign presentation
Kraft Mac and Cheese campaign presentationKraft Mac and Cheese campaign presentation
Kraft Mac and Cheese campaign presentation
 
How to Create a Social Media Plan Like a Pro - Jordan Scheltgen
How to Create a Social Media Plan Like a Pro - Jordan ScheltgenHow to Create a Social Media Plan Like a Pro - Jordan Scheltgen
How to Create a Social Media Plan Like a Pro - Jordan Scheltgen
 
Top 5 Breakthrough AI Innovations Elevating Content Creation and Personalizat...
Top 5 Breakthrough AI Innovations Elevating Content Creation and Personalizat...Top 5 Breakthrough AI Innovations Elevating Content Creation and Personalizat...
Top 5 Breakthrough AI Innovations Elevating Content Creation and Personalizat...
 

Destacado

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 

Destacado (20)

PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 

Search Marketing Thursday - Januari 2014 - Martijn Geerlings