4. But victory was never assured
Big-spending groups called Super-PACs were largely
supporting Romney
Obama was usually ahead in the polls, but the
advantage was narrow and volatile
7. The fundraising challenge
In 2008, Obama campaign raised $750 million
Would
not be enough in 2012
$750 million?
Not impressed.
8. The fundraising challenge
But fundraising was proving more difficult in 2012
than in 2008
President
less available for fundraising events
In early campaign, we saw average online donation was
half of what it had been in 2008
People were giving less, and less often
We had to be smarter and more innovative
9. Overview
A/B testing in Obama‟s digital department
Lessons learned
Don‟t
trust your gut
Foster a culture of testing
Invest in your team
The big picture: make it personal
14. Testing = listening to your audience
Nearly every email was tested, often on multiple levels
We started with message and subject line tests
4
messages, 3 subject lines were tested on every national
send (sometimes 6x3)
These tests went to about 20% of the list
After an hour, we would send the winner to the remainder
of he list
16. Example: Subject lines
Test sends
version
v1s1
v1s2
v1s3
v2s1
v2s2
v2s3
v3s1
v3s2
v3s3
v4s1
v4s2
v4s3
v5s1
v5s2
v5s3
v6s1
v6s2
v6s3
Subject line
Hey
Two things:
Your turn
Hey
My opponent
You decide
Hey
Last night
Stand with me today
Hey
This is my last campaign
[NAME]
Hey
There won't be many more of these
deadlines
What you saw this week
Hey
Let's win.
Midnight deadline
Each draft was tested with three
subject lines
One subject line would usually be
common across all drafts, to help
make comparisons across
messages
17. Example: Best vs. Worst Versions
Test sends
version
v1s1
v1s2
v1s3
v2s1
v2s2
v2s3
v3s1
v3s2
v3s3
v4s1
v4s2
v4s3
v5s1
v5s2
v5s3
v6s1
v6s2
v6s3
Subject line
Hey
Two things:
Your turn
Hey
My opponent
You decide
Hey
Last night
Stand with me today
Hey
This is my last campaign
[NAME]
Hey
There won't be many more of these
deadlines
What you saw this week
Hey
Let's win.
Midnight deadline
Full send (in millions)
donors
money
263
$17,646
268
$18,830
276
$22,380
300
$17,644
246
$13,795
222
$27,185
370
$29,976
307
$16,945
381
$25,881
444
$25,643
369
$24,759
514
$34,308
353
$22,190
273
263
363
237
352
$22,405
$21,014
$25,689
$17,154
$23,244
$4
$3
$2
$1
$0
ACTUAL ($3.7m)
IF SENDING AVG
IF SENDING
WORST
$2.2 million additional revenue from
sending best draft vs. worst, or $1.5
million additional from sending best vs.
average
18. Test every element
After testing drafts and subject lines, we would split
the remaining list and run additional tests
Example:
Unsubscribe language
Variation
Recips
Unsubs
Unsubs per
recipient
Significant differences in unsubs
per recipient
578,994
105
0.018%
None
578,814
79
0.014%
Smaller than D4
578,620
86
0.015%
Smaller than D4
580,507
115
0.020%
Larger than D3 and D4
19. No, really. Test every element.
We also were always running tests in the background
via personalized content
20. Tests upon tests upon tests
Review: Every piece of communication was an
opportunity to test
A single email often had many tests attached
Subject & draft tests
Full-list tests
Background personalization tests
21. The results
Campaign raised over one billion dollars
Raised over half a billion dollars online
Over 4 million Americans donated
Recruited tens of thousands of volunteers, publicized
thousands of events and rallies
Did I mention raising >$500 million online?
Conservatively, testing probably resulted in ~$200 million in
additional revenue
24. Don‟t trust your gut
We don‟t have all the answers
Conventional
wisdom is often wrong
Long-held best practices are often wrong
You are not your audience
There was this thing called the Email Derby…
If
even the experts are bad at predicting a winning
message, it shows just how important testing is.
25. Experiments: Ugly vs. Pretty
We tried making our emails prettier
That
failed
So we asked: what about ugly?
Ugly yellow highlighting got us better results
27. The culture of testing
Check your ego at the door
Use every opportunity to test something
Compare against yourself, not against your
competitors or “the industry”
Are
you doing better this month than last month?
Are you doing better than you would have otherwise?
28. When in doubt, test
In a culture of testing, all questions are answered
empirically
Example: With the ugly yellow highlighting, we worried
about the novelty factor
Maybe highlighting would only work for a short time before
people started ignoring it (or being irritated by it).
We decided to do a multi-stage test across three consecutive
emails
29. The ugly highlighting experiment
Experimental design:
First Email
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Second Email
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Third Email
Group 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Determined through this test that novelty was indeed a factor
30. Keep a testing calendar
On the Obama campaign we had short-term and longterm calendars for national emails
We added a “tests” column to plan out which tests would
be attached to which emails
If we saw blank spaces, it would remind us to think of
more tests to run!
Important to do frequent brainstorming sessions
31. Circulate your test results internally
We had an internal listserv entirely for the express
purpose of circulating test results
Helped get buy-in and increased familiarity with the
testing process
Prompted discussions and generated new ideas for
tests
33. OFA Digital Department
Grew from a small team in spring 2011 to a department of
200+ in 2012
Outbound (email, social, mobile, blog)
Ads
Front-End Development
Design
Video
Project management
Digital Analytics
34. Hire smart, diverse talent
Many Obama staffers (maybe most) had never worked
in politics before
Experience
less important than aptitude & passion
Diverse voices led to better content, analysis
Digital analytics: looked for people with strong
quantitative skills, willingness to learn
Staff
learned politics and programming on the job
36. Big data ≠ big brother
Testing allows you to listen to your user base
Let
them tell you what they like
Whether through A/B testing or behavioral segmentation,
optimization gives them a better experience
Usually, the interactions that are the most human are
the ones that win
37. Be human!
In general, we founds shorter, less formal emails and
subject lines did best.
Classic example: “Hey”
When we dropped a mild curse word into a subject line, it
usually won
“Hell yes, I like Obamacare”
“Let‟s win the damn election”
“Pretty damn cool”
38. Behavioral segmentation
Behavioral segmentation makes the experience personal
Donor vs. non-donor
High-dollar vs. low-dollar
Volunteer status
What issues do people say they care about?
After using A/B tests to create a winning message, we
could tweak it slightly for various behavioral groups and
get better results
39. Experiments: Personalization
Adding “drop-in sentences” that reference people‟s past
behavior can increase conversion rates
Example: asking recent donors for more money
…it's going to take a lot more of us to match
them.
Will you donate $25 or more today?
…it's going to take a lot more of us to match them.
You stepped up recently to help out -- thank
you. We all need to dig a little deeper if we're
going to win, so I'm asking you to pitch in
again. Will you donate $25 or more today?
Added sentence significantly raised donation rate
Confirmed in several similar experiments
40. Mobilization = Human Interactions
From 2012 Campaign Manager Jim Messina:
“My favorite story is from a volunteer in Wisconsin 10 days
out [from Election Day]. She was knocking on doors on
one side of the street and the Romney campaign was
knocking on doors on the other side of the street…”
41. Mobilization = Human Interactions
“… [The Obama volunteer] was asked to hit two doors. One
was an undecided voter and she knew exactly what to say.
The other was an absentee ballot and she was told to make
sure they filled it out and returned it. On the other side of the
street, the Romney campaign was knocking on every single
door. Most of the people weren‟t home, and most of the
people that were home were already supporting Barack
Obama. She looked at me and said, „You‟re using my time
wisely.‟ That‟s what data can do.”
- Obama 2012 Campaign Manager Jim Messina
