Friday, March 7, 2014

Tracking Pirate Metrics - Retention

This is the third post in our series on tracking Pirate Metrics using Just the Facts.  Our focus this time is on the Retention metric.

Retention is all about tracking whether the users we acquired in the previous "Registration" step are engaged - is the site sticky and do they return consistently?  This is really the measure of whether the product is delivering on its value proposition.

The metric for retention is different for each site and therefore needs to be specific to the site experience.  It could range from tracking repeat visitors who consume content or perform searches to tracking who clicks on the links in a site-generated email.

Let's take as our example Just the Facts itself.  The key value proposition is delivering your key metrics in a daily email, so the most logical retention metric would be tracking whether the emails are used or not.  For our purposes, we'll track the open rate of emails.  We'll assume that viewing the email is a good proxy for engagement, although you could also imagine a more sophisticated measure based on clicking through one of the email links or time spent reading the email.

As in our previous metrics, we'll use Google Analytics to actually capture this metric value.  We'll begin with a count of the emails opened each day using the Google Analytics event tracking capability.  Following the approach in this blog entry, we'll embed a tracking pixel in the body of the email that uses the GA API to register an "email open" event:

       <img src="http://www.google-analytics.com/collect?v=1&tid=${googleAnalytcisId}&cid=${user.id}&t=event&ec=email&ea=open&cs=notification&cm=email&cn=daily&el=daily&cm1=1" />

We can then subscribe to this metric from the Behavior > Events > Top Events > Overview dashboard by drilling into the "email" event category and subscribing to "Unique Events".  Let's assume that in this case we want to look at the email open rate on a day-to-day basis, so we choose "Yesterday" for the date range before subscribing:
Google Analytics Date Range Picker
and then activate the extension and choose the "Unique Events" metric:
Subscribe to Google Analytics Event
However, we're not quite done yet.  The problem with "Unique Events" by itself is that an increase in subscribers also increases the total events; it's not an independent metric that we can reliably use to measure the effect of experiments to improve retention, such as changing email frequency or writing more compelling headlines.

Instead, what we really want to track is the percentage of daily emails opened; an increase in this number is a reliable sign that more of our current subscriber base is engaged.

To calculate the percentage we also need to track the total number of daily emails sent out.  Since this is an event that happens on the back-end, it may be less convenient to track this using the Google Analytics tracking code.  One alternative is to generate a back-end event for each email sent using a statistic capturing service like StatHat, which provides APIs for a variety of back-end frameworks.

Here is an example of a Java call to report a count of the daily emails sent:

com.stathat.StatHat.ezPostCount(USER_KEY,  "Daily Report Email", 20);

Once data is being reported with this call, we can subscribe to the "1 month @ 1 day" Maximum value for this stat from the StatHat dashboard to get yesterday's result:
Subscribe to StatHat Metric
Now we're ready to calculate our "Daily Emails Opened" fact by selecting the two input facts and clicking the "Combine" button ():
Combining Two Facts
We'll divide one fact by the other, dragging the facts so they are in the right order and selecting the "Divided By" dropdown, and label the new fact "Retention (Daily Emails Opened)":
Combining Two Facts as a Percentage
Lastly, we can choose the two input facts, edit their settings, and unselect "Include in Daily Email" since we only care about monitoring the combined fact:
Turning "Include in Daily Email" Flag Off
We'll now receive a daily email that includes 3 of our 5 total pirate metrics:
Daily Report
The following post will look at adding in a Revenue metric by subscribing to a payment processing service's dashboard.

No comments:

Post a Comment