Notes on commonly missed aspects of designing an analytics strategy.
The first part of the presentation deals with well known lean-startup principles that many new practitioners seem (from my personal experience) to consistently overlook.
The second and third part deal with Churn and Customer Acquisition Cost, two important but not nice-looking metrics that startups seem to avoid or oversimplify because... they both "hurt".
Presentation delivered at Balkan Venture Forum 2015 in Zagreb.
How Generative AI Is Transforming Your Business | Byond Growth Insights | Apr...
The Art of Metrics
1. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
The Art of Metrics
notes about commonly missed aspects of
designing an analytics strategy
2. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
george@
useful.io
we build software products using Meteor.js
3. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
“Analytics” does not refer to a single business function or purpose.
It’s a multi-purpose toolkit for decision making and monitoring.
operations
ensure your business operates smoothly
troubleshooting / optimizing
why things happen? how can I improve?
predicting the future / investor pitch
will my start-up grow to…?
5. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
• Agenda:
• Part A: things people forget over and over
• Part B & C: things people avoid over and over
6. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Part A: Tips on picking
good metrics
7. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics depend on
your growth stage
Stage
EMPATHY
I’ve found a real, poorly-met need that a
reachable market faces.
STICKINESS
I’ve figured out how to solve the problem in a
way they will keep using and pay for.
VIRALITY
I’ve found ways to get them to tell their friends,
either intrinsically or through incentives.
REVENUE
The users and features fuel growth organically
and artificially.
SCALE
I’ve found a sustainable, scalable business with
the right margins in a healthy ecosystem.
Gate
Thefivestages
Source: Lean Analytics (by A.Croll & B. Yoskovitz) highly recommended reading
8. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics depend on
what you do
Empathy
Stickiness
Virality
Revenue
Scale
E-
commerce
SaaS Media
Mobile
app
User-gen
content
2-sided
market
Interviews; qualitative results; quantitative scoring; surveys
Loyalty,
conversion
CAC, shares,
reactivation
Transaction,
CLV
Affiliates,
white-label
Engagement,
churn
Inherent
virality, CAC
Upselling,
CAC, CLV
API, magic #,
mktplace
Content,
spam
Invites,
sharing
Ads,
donations
Analytics,
user data
Inventory,
listings
SEM, sharing
Transactions,
commission
Other
verticals
(Money from transactions)
Downloads,
churn, virality
WoM, app
ratings, CAC
CLV,
ARPDAU
Spinoffs,
publishers
(Money from active users)
Traffic, visits,
returns
Content
virality, SEM
CPE, affiliate
%, eyeballs
Syndication,
licenses
(Money from ad clicks)
A more information-rich version is available at: bit.ly/BigLeanTable
Source: Lean Analytics (by A.Croll & B. Yoskovitz) highly recommended reading
9. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics are actionable,
auditable, accessible
• Actionable: need to result to an action
• Auditable: can be replicated, accuracy can be
verified
• Accessible: can be measured, everyone
understands what they represent, everyone has
access
source: Eric Rees’ (The Lean Startup)
10. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics are “mindful”
• Qualitative versus quantitative metrics discover a story (*) vs
prove/confirm a story
• Vanity versus actionable metrics feel good vs changing your
behavior by helping you pick a course of action.
• Exploratory versus reporting metrics speculative insights vs
day-to-day operations health status.
• Leading versus lagging metrics predict the future vs explain
the past.
• Correlated versus causal metrics two metrics change
together(**) vs one metric causes another metric to change.
11. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics: are rates
0
50
100
150
200
April May June July August
Active users (x100)
0
50
100
150
200
April May June July August
0
500
1000
1500
2000
April May June July August
12. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Good metrics: are not only
about growth
Eric Rees’ (The Lean Startup)
“Sustainable growth is a requirement for any business. […]
New customers come from the actions of past customers”
The three growth engines:
Virality
Rate of growing
Stickiness
Rate of churn (compound)
Price
Rate of expenditure
14. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Churn
• understand how good you are at retaining
customers
• identify what actions result in a higher retention rate
15. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Churn
• users who abandoned (over a specific time frame)
• there’s more than one types of churn
16. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
–source: Localitics: http://info.localytics.com/blog/how-to-keep-your-app-users-with-the-3x3-rule
“58% of app users churn in the first 30 days
after they download your app, on average. And,
over the first three months, 75% of app users
will churn.”
17. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Churn: an oversimplified
formula
https://www.shopify.com/technology/4018382-defining-churn-rate-no-really-this-actually-requires-an-entire-blog-post
have you spotted the flaw?
(number of churns during period)
(number of customers at beginning of period)
18. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Churn in practice…
https://www.shopify.com/technology/4018382-defining-churn-rate-no-really-this-actually-requires-an-entire-blog-post
http://christophjanz.blogspot.de/2012/05/know-your-user-cohorts.html
19. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Predicting Churn
compounding
depends on user replacement
5% a month could be between…
46% - 80%… per year!
20. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Measuring Churn
• New customers will have a higher churn rate than
mature customers
• Measuring Churn will very likely force you to do
more fine-grained segmentation.
• Second reminder: make sure you factor-in user
acquisition… by modeling / calculating separately
http://sixteenventures.com/saas-churn-rate
21. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
“Churn will grow because your product market fit is
just now improving”
[as a result]
“Churn plagues early stage companies with volatile
product market fit, so assume the worst!”
source: Nikos Moraitakis: https://medium.com/@moraitakis/early-stage-saas-financial-planning-1f9e848f63ef#.7084b188y
22. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
–source: Bessemer Venture Partners via http://sixteenventures.com/saas-churn-rate
“[…] an acceptable [SaaS] churn rate is in the 5
– 7% range ANNUALLY, ”
23. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
–source: Fred Wilson (http://avc.com/2011/07/301010/)
“[Typically] 30% of the registered users/
downloads will use the service each month
10% of the registered users/downloads will use
the service each day
the max number of concurrent users of a real-
time service will be 10% of the number of daily
users[…]”
There is no single truth
24. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Part C:
Customer Acquisition Costs
(CAC)
25. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
–Anonymous
“Keeping a customer is [generally] cheaper
than getting a new one.”
26. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Customer Acquisition Costs
(CAC)
• The actual cost of growth (almost…)
• Crash course (*):
• Average Revenue Per User (ARPU): Revenues / Users
• Lifetime Value (LTV): ARPU * Lifetime
• Customer Acquisition Cost (CAC):
Total monthly acq costs / new monthly customers
27. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
–Bill Clark, Micro Ventures (http://microventures.com/education/metrics-every-startup-should-track)
“you have to take every dollar you spend on
reaching potential customers—from marketing
to public relations and beyond—and divide it
by the number of customers you actually
acquire in a given period”
28. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
CAC: making or breaking?
Are We Bleeding? CAC > LTV
Are We Happy(*)? 3 x CAC < LTV
29. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
• CAC(s) are not uniform; put effort into creating a
detailed model
• CAC will most likely grow* (but beware channel
capacity and growth rate ceilings)
30. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Breaking down CAC
source: Rob Moffat https://medium.com/venture-capital-growth-hacking/understanding-customer-acquisition-costs-74aec7538b4d
31. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
CAC considerations
• Focus on CPA (acquisition) not CPV (visit);
conversion rate varies dramatically per channel
• Use it as a driver to ask questions like:
“What can I do to grow volume of acquisitions through my free
channels?”
“What is likely to happen to CPA in my paid channels as I grow
spend? What limits will I hit?”
“Are there other marketing channels I should be looking at?”
“Start layering in paid acquisition, starting with the cheapest channels”
source: https://medium.com/venture-capital-growth-hacking/understanding-customer-acquisition-costs-74aec7538b4d
32. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Recommended further reading
(topics not covered in this presentation)
• On financial planning: “https://medium.com/@moraitakis/early-stage-saas-
financial-planning-1f9e848f63ef#.7084b188y"
— by Nikos Moraitakis
• On market sizing: “http://www.bothsidesofthetable.com/2009/06/16/sorry-guys-its-
the-size-of-the-wave-not-the-motion-of-the-ocean/“
— by Mark Suster
• On growth: “http://www.paulgraham.com/growth.html”
— by Paul Graham
• On the tech implementation of analytics: “https://segment.com/academy/“ &
“http://info.localytics.com/blog"
33. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Front-load your analytics strategy.
It will pay off.
34. George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015
Thank you
george@useful.io
george@voulgaris.net
@gevou
George Voulgaris, Ph.D. - The Art of Metrics
Zagreb, Dec 4th, 2015