TL;DR
My takeaways from the Duolingo reading journey covered articles from starting with A/B testing to their growth model and beyond aggregate metrics:
- Experiment with A/B testing and your customer journey; be careful to make a significant impact.
- There is more than retention rate, think about different users and their retention rate: new users, current users, resurected users, reactivated users.
- Duolingo set up a great product growth model for themselves by focusing on current users and its compounding effects. Increasing the Current User Retention Rate (CURR) 2% month-over-month had the largest impact on DAU (75% over three years).
- BUT: “Current Users has grown into a new monolith of users (90% of our DAU fall into this state!)” – you leave opportunities on the table AND optimize the system to serve one customer segment only.
- How can you address other segments beyond Current Users and optimize for them? How do you disaggregate metrics?
- Focusing and feeding Current Users might have a compounding effect on DAUs.
- Inspire yourself with experiments by other (best-in-class) industries, like for example Gaming or Uber.
- What is a meaningful metric? What might be a “movable” metric?
A start into tracking and metrics
During the last weeks I stumbled upon some great reads about Duolingo and product growth. I must admit, I’m feeling like a beginner in this area and therefore thought to write about it.
Especially when it comes to metrics, there is so much out there that makes me feel like a beginner:
At MagentaHome, unbounded weighted customer retention after 6 months is one of the chosen KPI. There are more retention possibilities out there, for example here at count.ly.
The Tenets of A/B Testing from Duolingo’s Master Growth Hacker by Gina Gotthilf
My Duolingo journey startet with this article which is and was very impressive from my POV by Gina Gotthilf: The Tenets of A/B Testing from Duolingo’s Master Growth Hacker. They ran some very impressive A/B tests to move ahead.
- Delayed Sign-Up: with the goal to let customers find the right time to register. Very powerful idea with creating soft walls.
- Streaks: one of the major challenges is getting people to come back into the app – user retention. So, one solution they ran into was inspired by games.
- Badges: rewards with meaningful accomplishments
- The In-App Coach
Overall, they run more than 70 A/B tests during her time when publishing the article. Key to consider:
- Rake in and rank continuously – there are so many ideas out there.
- How many people will be affected by this change?
- How many hours will go into running the test? Refine and repeat.
“For new apps, you should focus on A/B tests that bring a 20–30% increase in your metrics”.
Gina Gotthilf
How Duolingo reignited user growth by Jorge Mazal
Then a new post came across gaining my attention, which is the one from Jorge Mazal: How Duolingo reignited user growth. By mid 2018, they started to work on retention and grow DAU by 4.5x over four years – impressive number! They again had several phases:
- Increasing gamification – inspired by copying game patterns again.
- Referrals – inspired by copying Uber’s referral ideas.
- Using data and models – now this really got my attention as they introduced and differentiated various retention ideas.
- Current users retention rate (CURR): The chance a user comes back this week if they came to the product each of the past two weeks
- New users retention rate (NURR): The chance a user comes back this week if they were new to the product last week
- Reactivated user retention rates (RURR): The chance a user comes back this week if they reactivated last week
- Resurrected user retention rate (SURR): The chance a user comes back this week if they resurrected (from a longer absence) last week
With all bets and initiatives, the Current User Retention Rate and impact on DAU in three years increased by 75%!
There’s more gold in the article, enjoy reading!
Meaningful metrics: How data sharpened the focus of product teams by Erin Gustafson
The newest one I found is the one by Erin Gustafson is about Meaningful metrics: How data sharpened the focus of product teams.
Erin recapped a lot from the previous post of Jorge Mazal and was credited as well:
The results were clear cut: Increasing the Current User Retention Rate (CURR) 2% month-over-month had the largest impact on DAU. We staffed a team who started running A/B tests to see whether 1) CURR is a metric we can move and 2) moving CURR moves DAU …
The post comes with a cleaner illustration of their growth model:
The interesting question she raises after the recap:
“What opportunity are we leaving on the table by reducing a diverse learner base to a simple average?”
The interesting hypothesis behind the question and growth seems to be, that Current Users has grown into a new monolith of users (90% of our DAU fall into this state!).
Wow! Now this makes sense to me, as the system itself is optimized for Current Users only.
Another key takeaway of this article let me write this post:
What is a meaningful metric to me? What dimensions are influencing them? How can I have found out?
More interesting posts on Lenny’s blog – paid subscribers only
Lenny Rachitsky wrote another post about How Duolingo builds product, but it’s for paid subscribers only.
I also found another interesting post on b2c products and what metrics to use highly depends on your business model, which absolutely makes sense. Read more here: The most important consumer metrics to track
Unfortunately, I’m not a paid subscriber, so there’s not that much to add from here.
If you’ve came that far, thank you very much for reading! I’d love to learn more about you and your thoughts. Was this helpful? Would you like to exchange about that?
Happy to connect on your favorite channel.
All the best,
Patrick
Patrick Sauerwein | @virtual_patrick 