Everything you've learned about A/B Testing is based on the fundamentally flawed belief that there's one right answer. But the era of mass-market, one-right-answers is over. A/B Testing is our most valuable tool in the battle to create a more engaging web. But our strategy is broken. Don't worry, we can gain a better understanding of our users with a little data science. And we can reinvent A/B Testing... I will show you how. At Civis Analytics, we specialize in Data Science. From here, we can clearly see that all people are not the same. So why are A/B Tests designed to search for a single solution? In this session I'll show you where A/B Testing is headed next. See you in Austin!

1. Everything
you think about
A/B tes5ng
is wrong!
@chuparkoff #ABTestSX

2. @chuparkoff
VP of Product

4. Graham

5. Graham The control

6. Graham Dalton

8. UNDERSTAND ENGAGE
with the power of data science
REACH

9. Dan Wagner: What you can learn
from Data Science in Poli5cs
Monday
11:30am
ACC 9ab

10. A representa5ve sample of the talent at

11. Lived at the south pole
studying neutrino par5cles
A representa5ve sample of the talent at

12. Lived at the south pole
studying neutrino par5cles
PHD from MIT in the field
of network op5miza5on
A representa5ve sample of the talent at

13. Lived at the south pole
studying neutrino par5cles
PHD from MIT in the field
of network op5miza5on
Chief Analy5cs Officer for
the Obama 2012 campaign
A representa5ve sample of the talent at

14. Lived at the south pole
studying neutrino par5cles
PHD from MIT in the field
of network op5miza5on
Searched for the Higgs Boson
at the Large Hadron Collider
Chief Analy5cs Officer for
the Obama 2012 campaign
A representa5ve sample of the talent at

15. Lived at the south pole
studying neutrino par5cles
PHD from MIT in the field
of network op5miza5on
Searched for the Higgs Boson
at the Large Hadron Collider
Chief Analy5cs Officer for
the Obama 2012 campaign
Really likes
Lord of the Rings
A representa5ve sample of the talent at

16. Who here
is doing a lot of
A/B testing?
not muchsomea lot

17. Who here
is doing some
A/B testing?
not muchsomea lot

18. Who here
isn’t doing much
A/B testing?
not muchsomea lot

19. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

20. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

21. A/B testing basics
A/B tests help
you to find the
right answer

22. Which image is the most effective?
A/B testing basics
Which button drives the most sign ups?
Which feature drives the most engagement?
Which video persuades the audience?

23. A/B testing basics
Which button drives the most sign ups?

24. A/B testing basics

25. A/B testing basics

26. A/B testing basics
Try it free Try it freeTry it free

27. A/B testing basics
Try it free

28. A/B testing basics
Try it free Try it freeTry it free

29. A/B testing basics
Try it free

30. A/B testing basics
Try it free Try it freeTry it free

31. A/B testing basics
Try it free

32. A/B testing basics
Try it free Try it freeTry it free

33. A/B testing basics
Try it free
Many people think
the green button is the best.

34. A/B testing basics
They’re wrong

35. A/B testing basics
They’re wrong
…some5mes.

36. A/B testing basics
Which image is the most effective?

37. A/B testing basics

38. A/B testing basics

39. A/B testing basics

40. A/B testing basics

41. A/B testing basics

42. A/B testing basics

43. A/B testing basics

44. A/B testing basics

45. A/B testing basics
They’re wrong, too

46. A/B testing basics
They’re wrong, too
…some5mes.

47. A/B testing basics
But before I get into that…

48. A/B testing tools
A/B testing basics

49. A/B testing tools
A/B testing basics

50. A/B testing tools
A/B testing basics

51. A/B testing tools
A/B testing basics

52. A/B testing basics
All of these are
great for ‘click-races’.

53. not muchsomea lot
Who here
is doing a lot of
A/B testing?
But I’m going to show you
how you can take this further.

54. Who here
is doing a lot of
A/B testing?
a lota lota lot
But I’m going to show you
how you can take this further.

55. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

56. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

57. Three problems
Least common denominators
There is no single right answer
It’s about influence, not clicks
@chuparkoff #ABTestSX

58. A/B tests help
you to find the
right answer
Three problems There is no single right answer

59. d
ONE DOES NOT SIMPLY FIND
FIND THE RIGHT ANSWER

60. Three problems: There is no single right answer
Different people,
like different things.

61. Try it free Try it freeTry it free
Three problems: There is no single right answer

62. Three problems: There is no single right answer

63. Three problems: There is no single right answer

64. Three problems: There is no single right answer

65. Two problemsThree problems: There is no single right answer

66. Three problems: There is no single right answer

67. Three problems: There is no single right answer

68. Three problems: There is no single right answer

69. Three problems: There is no single right answer

70. Two problemsThree problems: There is no single right answer

71. Individual preference.
It’s a thing.
Three problems: There is no single right answer

72. Three problems
Least common denominators
It’s about influence, not clicks
There is no single right answer
@chuparkoff #ABTestSX

73. Let’s say you are the
Humane Society
Three problems: Least common denominators

74. Three problems: Least common denominators

75. Three problems: Least common denominators

76. Three problems: Least common denominators

77. Three problems: Least common denominators

78. Three problems:
kitten
OR
puppy
OR
lizard
Least common denominators

79. Kitten
AND puppy
AND lizard
Three problems: Least common denominators

80. We call this the
puppy - monkey - baby
problem
Three problems: Least common denominators

83. The least common denominator
of preference.
Three problems: Least common denominators

84. Three problems
Least common denominators
It’s about influence, not clicks
There is no single right answer
@chuparkoff #ABTestSX

85. Three problems It’s about influence, not clicks
Try it free Try it freeTry it free

86. Three problems It’s about influence, not clicks

87. Three problems It’s about influence, not clicks

88. Three problems It’s about influence, not clicks

89. Three problems It’s about influence, not clicks

90. Three problems It’s about influence, not clicks

91. Three problems It’s about influence, not clicks

92. Three problems
Least common denominators
It’s about influence, not clicks
There is no single right answer
@chuparkoff #ABTestSX

93. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

94. A/B testing basics
What should you do about it?
Three problems
Everything you think about A/B tes5ng is wrong!
@chuparkoff #ABTestSX

95. Your
audience
What should you do about it?

96. Golf &
tennis pros
Value
shoppers
Fitness
nuts
Outdoor
adventurers
Audience segments
What should you do about it?

97. Golf &
tennis pros
Value
shoppers
Fitness
nuts
Outdoor
adventurers
What should you do about it?

98. Golf &
tennis pros
Value
shoppers
Fitness
nuts
Outdoor
adventurers
What should you do about it?

99. If you’re doing ‘click-races’…
What should you do about it?

100. Value
shoppers
The
‘best answer’
only pleases
your biggest
group.
What should you do about it?

101. Those segments have differing value
Golf &
tennis pros
1.9x
Value
shoppers
0.3x
Fitness
nuts
1.1x
Outdoor
adventurers
1.4x
What should you do about it?

102. Data science gives us the ability to
understand customers
in a way we never could before
What should you do about it?

103. Who here
is doing a lot of
data science?
not muchsomea lot
What should you do about it?

104. Who here
is doing some
data science?
not muchsomea lot
What should you do about it?

105. Who here
isn’t doing much
data science?
not muchsomea lot
What should you do about it?

106. Data science will give you
segmenta5on at scale
What should you do about it?

107. Your
audience
What should you do about it?

108. Golf &
tennis pros
Value
shoppers
Fitness
nuts
Outdoor
adventurers
Audience segments
What should you do about it?

109. Golf &
tennis pros
Value
shoppers
Fitness
nuts
Outdoor
adventurers
What should you do about it?

110. UNDERSTAND REACH ENGAGE
What should you do about it?

111. Survey your users.
Ask them lot’s of questions.
What should you do about it?
UNDERSTAND

112. Measure where they came from
& where they go.
What should you do about it?
UNDERSTAND

113. Create user segments
based on product usage.
What should you do about it?
UNDERSTAND

114. Learn how to do some
basic predictive modeling.
What should you do about it?
UNDERSTAND

115. Find your segments
What should you do about it?
UNDERSTAND

116. UNDERSTAND REACH ENGAGE
What should you do about it?

117. Talk to different segments
over different channels
What should you do about it?
REACH

118. Create segment-optimized
landing pages
What should you do about it?
REACH

119. Use segment-optimized
media targeting
What should you do about it?
REACH

120. UNDERSTAND REACH ENGAGE
What should you do about it?

121. Match the right content
to the right segments.
What should you do about it?
ENGAGE

122. Avoid showing detrimental content
to the wrong segments.
What should you do about it?
ENGAGE

123. What should you do about it?

124. UNDERSTAND REACH ENGAGE
What should you do about it?
If you get stuck, we can help.

125. Two roads diverged in a wood and I —

126. Two roads diverged in a wood and I —
I did A/B tes4ng to see which one
had higher click-throughs,
and that has made all the difference.
@odannyboy