Wednesday, April 2, 2014

Tracking Pirate Metrics - Targets and Warnings

We've now completed our metrics dashboard in Just the Facts by tracking the five components of Pirate Metrics.  However, in its present state the dashboard is not very actionable.  There's no easy way to know whether or not we're making progress towards our goals, and it's easy to miss a change in a metric's value that indicates a problem that requires investigation.

Let's address this by adding targets and warnings to our facts.  A target is a desired value; we can track our progress towards the target over time and be notified when we reach it.  A warning is an alert that is triggered when certain conditions are met, such as a fact going outside a target range of values.

In our case, let's set two warnings and one target as follows:
  • Acquisition (Unique Visitors) - Set a range of values.  If acquisition goes below the minimum, we want to investigate why our traffic for the week was so low.  On the other hand, if acquisition goes above the maximum, that means we had a particularly large number of visitors, and it's worth investigating to see what the source of traffic was and how we can repeat it.
  • Retention (Daily Emails Opened) - Set a minimum value.  If an unusually low number of daily emails were viewed, it could indicate a problem with the email delivery or formatting that requires corrective action.
  • Revenue (Daily Sales) - Set a target value, with the expectation that sales increase over time and approach the target.

We apply the same approach for each fact, modifying it by clicking on the circle before the fact's current value:
Set Fact Warnings
To set the range for Acquisition, we click "Stay within" and enter values into the minimum and maximum fields:
Stay Within a Range

After saving our changes, note that the circle turns green to indicate that the fact is currently on target:
On Target Fact
 To set the minimum value for Retention, we again click "Stay within" but only provide a minimum value.  Since the raw value for this fact is a number between 0 and 1, we enter ".25" to specify that the percentage value should stay above 25%.
Stay Above a Value

And for Revenue, we fill in the Target field:
Set a Target Value

After saving our changes, note that the sparkline for this fact now graphs the target value:
Fact with Target
Whenever one of the facts goes out of bounds, we will receive an email with the details.  We will also be emailed when a fact reaches its target.  
Target Reached Email

To change this behavior, we can click on the gear icon on the top of the page and modify our email settings:
Settings Dialog

Over time, we may want to adjust our warning and target settings - once a target is reached, it may be time to set a new target, or simply remove the target and focus attention on a different metric.  As metric baselines change over time, the appropriate range or minimum values may also change.  It is easy to change the settings; simply click the same "Set Warnings" icon used to set the original values.