Wednesday, March 19, 2014

Tracking Pirate Metrics - Referral

This is the fifth post in our series on tracking Pirate Metrics using Just the Facts.  Our focus this time is on the Referral metric.

Referral is typically used to track the virality of your product, that is, the extent to which your existing users are driving your growth by referring new users.


As with retention, the best measure of referrals is always going to be specific to your product.  Some of the possible measures include:
  • the number of invites sent by existing users
  • the number of signups that can be attributed to the referral
  • the viral coefficient
  • the ratio of new customers to existing customers
  • cycle time, how much time it takes for a user to invite others.
Again using Just the Facts as an example, the feature of sharing a dashboard with colleagues is one obvious source of viral growth that we can track as a referral metric.

For this example, we'll use a different analytics web service.  Mixpanel is particularly useful for tracking fine-grained events, segmenting them on a variety of properties, and analyzing funnels of events.  We'll assume that Mixpanel events are already being used to track two key events related to referrals:
  •  an existing user sharing a dashboard
  •  a new user registering in order to view a dashboard that was shared with them.
To subscribe, we'll navigate to the "Trends" dashboard in Mixpanel, select the "Total visitors" dropdown value and activate the extension.  We'll subscribe to the "Share Dashboard" and "Referral Registration" facts: 
Subscribing to Mixpanel Facts
and then provide our account's API and Secret keys, available from the "Projects" tab under your account information.
Mixpanel Account Information Dialog
The system will subscribe to the total count of visitors from yesterday who performed the event in question, perfect for tracking referral metrics on a daily basis.

As with our previous facts, once we have created the subscription, we'll re-label the new facts to make it clear that they are Referral metrics:
Editing the Fact Label
Our complete dashboard, showing both source and calculated facts for all 5 categories of Pirate Metrics, now looks like this:
Dashboard
and the daily email report looks like this:
Daily Email
Our next post will look at setting targets and warnings on our facts to help us track progress and identify when facts need attention.

Tuesday, March 11, 2014

Tracking Pirate Metrics - Revenue

This is the fourth post in our series on tracking Pirate Metrics using Just the Facts.  Our focus this time is on the Revenue metric.

Tracking revenue tells you whether you are making money from your retained, engaged users.  There are a variety of cloud services that can assist with processing payments and tracking revenue; for our purposes, we will assume the use of Stripe for payment processing.

The Stripe dashboard displays a summary count of total customers and sales volume; we can capture this data and use it as the basis for a calculated fact that will track daily revenue.

Go to your Stripe dashboard and activate the Just the Facts plug-in, subscribing to the "Total volume" metric:
Subscribing to Stripe Total Volume
The subscription process will ask you to provide your secret key, which can be found under your account settings:
Stripe Secret Key Form
While we're now capturing the latest value of Total Volume every day, it is not in and of itself the metric we want to track.  Total Volume is a typical example of a "vanity metric" - it can only ever go up.  Instead, let's actually track the sales for a single day - if this is consistently increasing, it shows actual improvement in our business.

To track the day-to-day sales, we'll create a Combined Fact that calculates the difference between consecutive values.  Select "Total Volume" and click the "Combine" button:
Creating a Combined Fact
Select the "Difference between current and previous" radio button and label this fact "Revenue (Daily Sales)":
Calculating the Difference
As in our previous steps, we'll also want to edit the "Total Volume" fact to deselect from our daily email.

At this stage, we now have a dashboard tracking four out of five of our pirate metrics:
Dashboard Showing Source and Calculated Facts
Note that by default all facts, both source and calculated, are displayed.  Since some of the source facts are not of interest and are just used as the basis for calculations, we can bring up a more concise view by using the Filter dropdown to display just the Google Analytics and Combined facts:
Filtering Facts
For our next blog posting, we'll look at pulling in the final Pirate Metric, Referrals, from an additional analytics system.

Friday, March 7, 2014

Tracking Pirate Metrics - Retention

This is the third post in our series on tracking Pirate Metrics using Just the Facts.  Our focus this time is on the Retention metric.

Retention is all about tracking whether the users we acquired in the previous "Registration" step are engaged - is the site sticky and do they return consistently?  This is really the measure of whether the product is delivering on its value proposition.

The metric for retention is different for each site and therefore needs to be specific to the site experience.  It could range from tracking repeat visitors who consume content or perform searches to tracking who clicks on the links in a site-generated email.

Let's take as our example Just the Facts itself.  The key value proposition is delivering your key metrics in a daily email, so the most logical retention metric would be tracking whether the emails are used or not.  For our purposes, we'll track the open rate of emails.  We'll assume that viewing the email is a good proxy for engagement, although you could also imagine a more sophisticated measure based on clicking through one of the email links or time spent reading the email.

As in our previous metrics, we'll use Google Analytics to actually capture this metric value.  We'll begin with a count of the emails opened each day using the Google Analytics event tracking capability.  Following the approach in this blog entry, we'll embed a tracking pixel in the body of the email that uses the GA API to register an "email open" event:

       <img src="http://www.google-analytics.com/collect?v=1&tid=${googleAnalytcisId}&cid=${user.id}&t=event&ec=email&ea=open&cs=notification&cm=email&cn=daily&el=daily&cm1=1" />

We can then subscribe to this metric from the Behavior > Events > Top Events > Overview dashboard by drilling into the "email" event category and subscribing to "Unique Events".  Let's assume that in this case we want to look at the email open rate on a day-to-day basis, so we choose "Yesterday" for the date range before subscribing:
Google Analytics Date Range Picker
and then activate the extension and choose the "Unique Events" metric:
Subscribe to Google Analytics Event
However, we're not quite done yet.  The problem with "Unique Events" by itself is that an increase in subscribers also increases the total events; it's not an independent metric that we can reliably use to measure the effect of experiments to improve retention, such as changing email frequency or writing more compelling headlines.

Instead, what we really want to track is the percentage of daily emails opened; an increase in this number is a reliable sign that more of our current subscriber base is engaged.

To calculate the percentage we also need to track the total number of daily emails sent out.  Since this is an event that happens on the back-end, it may be less convenient to track this using the Google Analytics tracking code.  One alternative is to generate a back-end event for each email sent using a statistic capturing service like StatHat, which provides APIs for a variety of back-end frameworks.

Here is an example of a Java call to report a count of the daily emails sent:

com.stathat.StatHat.ezPostCount(USER_KEY,  "Daily Report Email", 20);

Once data is being reported with this call, we can subscribe to the "1 month @ 1 day" Maximum value for this stat from the StatHat dashboard to get yesterday's result:
Subscribe to StatHat Metric
Now we're ready to calculate our "Daily Emails Opened" fact by selecting the two input facts and clicking the "Combine" button ():
Combining Two Facts
We'll divide one fact by the other, dragging the facts so they are in the right order and selecting the "Divided By" dropdown, and label the new fact "Retention (Daily Emails Opened)":
Combining Two Facts as a Percentage
Lastly, we can choose the two input facts, edit their settings, and unselect "Include in Daily Email" since we only care about monitoring the combined fact:
Turning "Include in Daily Email" Flag Off
We'll now receive a daily email that includes 3 of our 5 total pirate metrics:
Daily Report
The following post will look at adding in a Revenue metric by subscribing to a payment processing service's dashboard.

Wednesday, March 5, 2014

Tracking Pirate Metrics - Activation

Following on our last post, we will now add Activation tracking to our daily email report.

Activation is all about turning your website visitors into users of your product.  Typically this involves completing a sign up process, although depending on your product it could also refer to an anonymous user successfully completing a key use case or spending a certain amount of time engaged with the site.

Let's assume we want to use number of completed registrations as our activation metric.  Let's also assume we're using Google Analytics for tracking, having set up a custom goal using the built-in "Create an account" template found in the administration screens:
Google Analytics Goal Setup
We'll name the goal "Register" and track it based on our registration confirmation screen:
Google Analytics Goal Details
Now, whenever a user reaches the /register/register relative link on the site, it will be tracked as a completed goal.

The subscription process is very similar to our earlier subscription to Google Analytics. In this case we start with Google Analytics' Conversions > Goals > Overviews dashboard and subscribe to the goal we are interested in:
Subscribing to a Google Analytics Goal
After the subscription is completed, we'll again re-label the fact and update its frequency:
Fact Settings Dialog
At this point our in-progress dashboard looks like this:
Dashboard with Two Facts
The following post will look at adding in a Retention metric by calculating it from a combination of a Google Analytics metric and our own custom data.

Tuesday, March 4, 2014

Tracking Pirate Metrics in Just the Facts

One of the most well-known frameworks for startup key performance indicators is Dave McClure's Pirate Metrics.  It defines five categories of metrics that correspond to the stages a web site's users pass through:
  1. Acquisition
  2. Activation
  3. Retention
  4. Revenue
  5. Referral
Because these metrics are captured and tracked in a variety of different systems of record, it's a natural fit to use Just the Facts to collect them into a single daily report.  This series of posts step through one example of how to define and consolidate these metrics.  At the end, we'll have a single, daily summary of these key metrics with built-in alerting:
Pirate Metrics Daily Report
In the process, we'll see how it's necessary to put a fair amount of thought into the appropriate metrics to track to ensure that you're targeting actionable rather than vanity metrics.  We'll also see how Just the Facts' ability to combine together and calculate metrics makes it such a powerful tool for reporting on metrics that would otherwise require custom engineering work.


Acquisition


Acquisition tracks the users arriving at your site; they have to become aware of your service before you have the opportunity to present your value proposition and convert them from visitors into users.  Typical measures in this category are the number of users visiting your site and the percentage of users that do not bounce; that is, users who have some degree of engagement with the site and are therefore potentially in the target audience.

In our example, let's assume we're using Google Analytics for basic tracking of visitor behavior.  Let's also assume that our key starting acquisition metric will be the weekly count of Unique Visitors.  We choose a weekly total to smooth over any day-to-day differences over the course of a work week, and we'll assume that a single day's bump in traffic is not a significant event for our long-term tracking purposes.

To subscribe, we log into Google Analytics using Google Chrome and from the default Audience > Overview dashboard select a date range of "Last Week:"
Google Analytics Date Range Picker
We then activate the Just the Facts Chrome extension and select the Unique Visitors metric:
Subscribe to Google Analytics
After giving permissions to Google Analytics:
Permissions Dialog
the new fact is added to our dashboard:
Newly Subscribed Fact
In order to provide a more descriptive label for this fact, we click the Settings icon () to bring up a dialog allowing us to re-label the fact.  We'll apply a label to make it clear that this metric is associated with acquisitions.  Since the value is only going to change once a week, we'll also update the Refresh Frequency to the beginning of the week:
Fact Settings Dialog
We now have our first of the pirate metrics tracked and emailed to us every day:
Daily Report showing Acquisitions
The following post will look at adding in the "Activation" metric using a custom Google Analytics goal.