2 - Defining Your Metric

Prévia do material em texto

01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 1/24
Defining Your Metric
In this lesson, we're looking into how to measure and analyze retention for our product. There are three key
steps that we need to go through to define our retention metric:
1. Frequency
2. Core behavior
3. Who
 
 
menu Previous Section (/c/2018-fall-growth-
series/retention-and-engagement/measuring-
analyzing/impact-of-incorrect-retention-metric)
Mark Complete & Next Section
(/c/cms_sections/839/mark_complete_and_nex
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/impact-of-incorrect-retention-metric
https://program.reforge.com/c/cms_sections/839/mark_complete_and_next_section
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 2/24
  
In our Reforge Retention & Engagement Diagnosis Sheet, we're going to be filling out the retention part of the
sheet over the next couple of lessons. Let’s dive right in!
 
 
Step One: Frequency
 
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 3/24
 
Key Question: What is the natural frequency in which the user
experiences the problem? 
 
 
We've already defined that as part of our use case map development, but at this step, we want to validate that
qualitative hypothesis with quantitative data. 
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 4/24
 
We’re going to do that with a histogram of usage. On the Y axis we have the number of users, and on the X axis,
we have the number of days they were active over a certain time period — in this case, the last 28 days.
 
 
We're going to follow several steps to confirm our frequency hypothesis:
1. Select a use case
2. Create a frequency histogram
3. Analyze the distribution
Nicole Bach
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 5/24
 
Step One: Select a use case
 
 
Let's go back to our Pinterest example and use everybody from the browsing use case where we have a natural
frequency of weekly. 
 
 
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 6/24
We'll take all the users who are more than 28 days old and have activated and completed the core action at
least once in the last 28 days. 
 
 
What we're looking for here are those people who have activated successfully. We want to weed out those that
have never activated and really hone in on the people who have gotten to understand the core value
proposition. 
 
Step Two: Create a frequency histogram
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 7/24
 
We're going to plot the users who completed the core action in the last 28 days in our histogram. 1,400 users
were active 23 days out of the last 28 days, each bar in the histogram showing how many users were active
that many days out of the last 28 days. 
 
 
Step Three: Analyze the distribution 
 
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 8/24
 
We expect to see certain patterns based on the qualitative hypothesis of our natural frequency. If our natural
frequency hypothesis is daily, we should be seeing a pattern where a spike occurs in the frequency histogram
around 21 to 25 days out of the month. 
 
 
That tells us that a huge portion of our users are using it daily at least 20 days out of the last 28. If our
hypothesis is weekly, we’d see the frequency histogram spike around five, six, or seven days out of the week. 
 
Nicole Bach
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 9/24
 
Congregation around this mark naturally indicates a weekly behavior, while for those that have a monthly
frequency, we would expect to see the histogram congregate to the far left, showing only one or two active
days out of the last 28 days. 
 
 
Now that we have our histograms plotted out, we can play with different components of their frequency. If we
have a longer natural frequency in our use case, we're going to want to extend the X-axis further, looking at a
longer time horizon to understand how usage behaves in the future.
If we do this quantitative analysis, we have to ask the question: What happens if our quantitative analysis
differs from our qualitative hypothesis? 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 10/24
 
 
As an example, we may know that our natural frequency is weekly, but the frequency histogram shows that it's
actually only occurring monthly. 
 
This is a hard truth and clarifies that a delta exists between the actual product you have and the product you
wanted to build based off of the problem definition that you're solving. You’ll either have to work on the product
to bring it to the point where it's solving the natural frequency of the problem, or go back to the qualitative
definition and make sure that definition is correct. 
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 11/24
Step Two: Core Behavior
 
 
Once we have the frequency identified, we can move on to understanding the core behavior of our retention
metric. 
Key Question: What action indicates we're delivering value to
the user? 
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 12/24
 
In other words, are we solving the problem or not? We're looking for an action, a signal from the user that
indicates that we are successfully solving the problem and the why. Our core action hypotheses should stem
from this problem and this why.
 
 
In the Pinterest example, we might come up with a number of hypotheses around actions like viewing the feed,
pinning or re-pinning something, or clicking on something in the feed. We think these might indicate that we're
actually solving their problem, delivering real value around solving their boredom and helping them find things
around their interests. 
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 13/24
 
In the HubSpot Sales efficiency use case — spending countless hours on inefficient activities like copying and
pasting emails — we might be looking at hypotheses like tracking an email, a meeting schedule, or maybe a
combination of these actions.
We then want to take these qualitative hypotheses of our core actions and do the exact same thing that we did
with frequency — validate it with quantitative data — this time by following several different steps. 
 
 
In order to confirm the core action hypothesis, we must:
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 14/24
1. Form groups that successfully do the action for successive periods
2. Create a cohort chart for different action hypotheses
3. Analyze retention by comparingStep One: Form groups of users that successfully did the action in a hypothesis for a
successive number of periods
 
Let's take a look at how this works across a few different examples. If our hypothesis is viewing the feed and
our natural frequency is weekly, we'll start by taking a group of users who viewed the feed for four successive
weeks. 
If our hypothesis was pinning or re-pinning and the natural frequency is weekly, we're going to look at users
who had at least one pin for four successive weeks. 
 
Nicole Bach
Nicole Bach
Nicole Bach
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 15/24
 
Step Two: Create a cohort chart for our different action hypotheses
 
 
We’ll show you how to create retention curves in a future section so we won’t go into depth here, but we do
want to quickly discuss what you should be able to see.
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 16/24
 
 
Once you plot the cohorts out on the retention curves by different actions, you’ll see that top line is the
retention curve for people taking the pinning action. The second line is for those clicking, and the third line is
those that are viewing.
We’re then ready for the next action.
Step Three: Analyze the retention curves
 
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 17/24
By comparing the patterns of the curves we find that our core action hypothesis of pinning actually shows a
much healthier retention pattern. Not only is it retaining a much larger amount of users, but it starts to flatten
off near the end, reinforcing that this is probably a winning hypothesis.
 
 
When we compare that to just viewing the feed, we find that the core action is not actually creating as many
long-term retained users. So choosing pinning as the core action in our retention metric would be better in this
case. 
Once we have the core behavior, we can move on to the last step.
Step Three: Who
 
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 18/24
 
To finish up the last component of defining our retention metric we need to ask one more question.
Key Question: Who is included in this metric? 
 
 
Here, we’re trying to add clarity to our retention metric and brand it internally so that everyone in the
organization understands it. When someone says, "Our retention metric is weekly active users," it starts to
prompt a lot of different questions. 
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 19/24
What's included in a user? 
What action did they have to do? 
If these aren’t answered, it's unclear who the user is and what the core action behind active is.
 
 
We should focus on the core action of the use case to brand the who part of this retention metric. 
 
 
Nicole Bach
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 20/24
There are two things we need to look at. First, the who, or the persona. Then, we look at the core action, re-
pinning. Those two things combined will help us determine the final component of our retention metric. 
In the Pinterest example, we find it in the browsing interest of weekly active re-pinners. That’s much more clear
than weekly active users because we know what action they're taking as part of that active definition. Similarly,
if we were to set a retention metric around planning a project, we might look at daily active re-pinners. 
 
 
Once we've gone through all the steps we might end up with various retention metrics. For Instagram, we
would end up with something like a daily active user viewing the feed, or for  Pinterest, a weekly active re-
pinner. 
 
Nicole Bach
Nicole Bach
Nicole Bach
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 21/24
 
But if we look at Airbnb for example, we’d end up with yearly active guests. 
 
 
We're combining the natural frequency of yearly active which equals at least one booking within that time
period, and we're branding the who part of that metric a guest. Similarly, on the host side, we might be looking
at weekly active hosts.
An e-commerce example like Blue Bottle Coffee has two different use cases. One is buying via a la carte,
solving the problem where a customer wants their choice of blends, origins, and specialty roasts. The other is a
subscription member, a Blue Bottle loyalist, who wants to make sure they never run out of Blue Bottle Coffee in
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 22/24
their home. 
 
 
The a la carte use case might end up with a retention metric of quarterly active buyers. They're certainly not
going to be buying on a daily or a weekly basis, but we find the natural frequency of quarterly and combine that
with a core action of making at least one purchase in branding that as a buyer. 
With the subscription member, we might end up with monthly active subscribers. They get the subscription on
a monthly basis, active if they've received a subscription and our who is a subscriber part of that equation. 
In the next lesson, we're going to talk about how we take these defined retention metrics and start to evaluate
our retention using cohorts. 
 
Application Question
Create a hypothesis for your retention metric using your natural frequency, core
behavior, and persona hypothesis. Then, explain your rationale behind each component
of the retention metric.
 
 
SUBMIT
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 23/24
Application Question
You’ve been dropped in as VP of Growth at Loom (https://www.useloom.com/). Loom
is a cloud-based video tool that allows you to easily record and share videos. With one-
click, you can start recording, then share to slack, email, etc. with another click. Loom
is used by over 400,000 teams, who use the tool for a variety of reasons, such as
replacing writing long emails, product demos and walkthroughs, showing aspects of a
product for sales or customer support reasons, and more.
 
 
Your goal is to dive into defining and analyzing retention for Loom. You want to get an
initial understanding of retention and identify immediate next steps.
 
What would you choose as Loom’s retention metric?
 
Hint: Think through the natural frequency, core action, and “who” is supposed to receive
value from the product.
 
 
 
SUBMIT
Weekly Active Recorders
Monthly Active Teams
Daily Active Users
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/impact-of-incorrect-retention-metric
https://www.useloom.com/
01/10/2018 Reforge
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/defining-your-metric# 24/24
Previous Section (/c/2018 fall growth
Copyright © 2018 REFORGE. Sharing of content and user accounts is strictly prohibited.
https://program.reforge.com/c/2018-fall-growth-series/retention-and-engagement/measuring-analyzing/impact-of-incorrect-retention-metric
https://program.reforge.com/c/cms_sections/839/mark_complete_and_next_section