“In God we trust, all others must bring data”. Intuition, experience and well known patterns may give us good indications of successful ideas and features, but nothing gets closer to the truth than data analysis and A/B testing. In this workshop, we’ll show how we do experimentation at Booking: what we test, how to get data through templates and JavaScript, and how we analyse the resulting metrics. We’ll live-code examples, see all potential caveats of dealing with the user tracking on the client-side, and show existent tools you can use to test your own ideas.

1. A/B testing workshop
“In God we trust, all others must bring data”
JSConf Colombia Workshop 2015

2. @shiota
github.com/eshiota
slideshare.net/eshiota
eshiota.com

5. A/B

6. A/B tests measure how a new idea
(version B/variant/test) performs
agains an existing implementation
(version A/base/control).

9. Buy now Buy nowversus

10. coin flip
Buy now
Buy now
50%
50%

11. When the user sees or is affected by
the idea, they are tracked and
become part of the test.

12. Buy now
Buy now
track(my_experiment)

13. Data about the website is generated
as users browse through pages and
do their tasks.

14. product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…

15. When there’s enough information to
make a decision, you can either stop
the test (keeping version A) or choose
version B, directing all traffic to it.

16. Buy now Buy now
Duration: 14 days
Visitors: 45.140 (22.570 per variant)
339 (1.5%) 407 (1.8%)
20% up
144.500 COP 147.390 COP
2% up
Number of purchases:
Average price:

17. coin flip
Buy now
Buy now
50%
50%

18. B
Buy now
100%

19. "But my design is obviously more
beautiful and intuitive than what
we have now! Why should I run
an A/B test?”
— the majority of designers

20. Quiz time!(prizes included)

21. A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

22. Reduced bounce rate in 1.7%

23. A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

24. Which performed better?
Increased CTR in 203%

25. A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

26. Which performed better?
43.4% more purchases

27. A: Raise your left hand B: Raise your right hand
Neutral: Don’t raise your hands
Which performed better?

28. Both were statistically equivalent
Which performed better?

29. Intuition
vs
Historical Analysis
vs.
Experimentation

30. We have a 2/3 chance of being
wrong when trusting our intuition.

31. People behave differently each
season/month/day of the week.

32. Different cultures lead to
different patterns of usage.

33. Data analysis alone provides
correlation but not causation.

35. Running your A/B test
(in 5 simple steps)

36. Step 1: Hypothesis

37. Analyse all possible inputs to come up
with an hypothesis to work on.

38. • Usability research
• Benchmarking
• Surveys
• Data mining
• Previous experiments

39. Hypothesis:
“If users from South America
countries relate more to the
website, they will book more.”

40. Step 2: Idea

41. Idea:
“If we add the country’s flag next
to the website’s logo, users will
relate more to the brand.”

44. Step 3: Setup

45. • Who will participate?
• What is the primary metric?
• Any secondary impacts?
• How will it be implemented?

46. • Users from Argentina, Bolivia, Brazil, Chile,
Colombia, Ecuador, Guyana, Paraguay, Peru,
Suriname, Uruguay and Venezuela, on all
platforms
• Conversion (net bookings) uplift is expected
• We expect more returning customers

47. <h1 class="main-header__logo logo">
<% if user.is_from_south_america && track_experiment(:header_flag_for_south_america) == "b" %>
<span class="main-header__logo__country-flag">
<%= user.country %>
</span>
<% end %>
<%= image_tag "logo.png" %>
</h1>

48. Step 4: Monitoring

49. Keep checking the metrics to see if
anything’s terribly wrong.

50. Avoid checking too often, let your test
get enough users and enough runtime.

52. Step 5: Data, decisions,
and next steps

53. When you reach the expected runtime,
number of visitors or effect, look at the
data and take a decision.

54. product added to cart
number of products added
purchase finished
average price per purchase
number of products seen
user has logged in
used guest checkout
customer service calls
…

55. Optimizely dashboard

56. • How were the primary and secondary
metrics impacted?
• What were the results isolated by each
country?
• What were the results isolated by each
language?
• Did any particular platform (desktop,
mobile devices, tablets) perform better?
• Was the impact on returning customers
any higher than first time visitors?

57. Based on the gathered data,
plan for next steps.

58. • Should we add a copy to the flag?
• Should we add a tooltip to the flag?
• Should we increase/decrease the flag size?
• Should we restrict it just for desktop users?
• Should we try this for a single country, or
other countries?

59. What can you test?

60. (almost) Everything.

61. You can test a small
design change.

62. versus

63. You can test large
design changes.

64. versus

65. You can test different copies.

66. versus
Submit
Book now

67. You can test technical
improvements and measure page
load time, repaints/reflows, and
conversion impact.

68. versus
jQuery 1.11.3
jQuery 2.1.3

69. You can even test back-end
optimisations and measure
page load time, rendering time,
CPU and memory usage etc.

70. if track_experiment(:my_optimized_query)
@users = my_optimized_query
else
@users = do_the_normal_thing
end

71. Live coding
(I hope that works.)

72. Find the code at:
https://github.com/eshiota/ab_workshop
Additional links:
https://www.optimizely.com/
https://github.com/splitrb/split/
http://whichtestwon.com
http://unbounce.com/
http://blog.booking.com/hamburger-menu.html
http://blog.booking.com/concept-dne-execution.html
Gracias!