Forget the North Star Metric: Here's What to Look at Instead

Many clients come to us with a specific problem. They have identified the key metric of their product, and the data shows an improvement, but something doesn't add up:

• Other metrics are getting worse
• Positive ratings are dropping
• Users aren't using the product as expected.

Why does this happen? Often, the reason is precisely the way the North Star Metric is treated. But first, let's take a closer look at what we're talking about.

What the North Star Metric is

The North Star Metric is a measure tied to the main value delivered to the user, that is, what the user finds good about your digital product.

Its purpose is to predict the long-term success of a product. That's exactly why its name recalls the North Star: just as the star lets you find your bearings and set a direction, our metric serves to guide the whole team toward the same goal.

The main characteristics of the NSM are that:

• It's about the user, not the product or the company that distributes it; revenue, for example, is not a good North Star Metric.
• It can be directly influenced; so it doesn't depend solely on external variables such as market trends.
• It tends to grow or decline; therefore, neither the number of users who have completed sign-up, which cannot decrease, nor the percentage of users active every day for at least 30 days, which can reach 100% at most, are good metrics.

So what are good examples of a North Star Metric? Some examples are the number of nights booked for Airbnb, or the number of minutes listened for Spotify.

Don't confuse it with Vanity Metrics

The North Star Metric is not a Vanity Metric. Vanity metrics, instead, are numbers that make you look good. They can even be useful, for example to keep team morale high or to convince investors, but they're no use at all for making decisions. In short: they look great on a slide, but they don't give you any concrete direction.

Examples of vanity metrics include:

• Views of the product landing page
• Total revenue from in-app purchases
• The number of social media followers

We also talked about this in our article on how to validate an app: the number of people signed up to a waitlist or the comments received when building in public are not the same as actual use of the product.

Paying attention to these metrics can make you lose focus on what really matters, that is, what actually determines success over time. It can also create a false sense of security about how your product is doing.

Even the giants fall for it: a real case

It's not only small companies that risk choosing the wrong metric. YouTube, too, has had to face the same problem over time. An article on its official blog, in fact, explains that in the early days YouTube's recommendations were based on the number of clicks on each video.

In 2011, the company noticed a problem. Suppose a user was searching, to use the original example, for the highlights of a Wimbledon match. The user would be shown some videos that, judging by title and thumbnail, seemed to show scenes from the match.

Clicking on them, they would sometimes discover that the video didn't show the players in action at all, but rather some fans commenting on the sporting events. Since that wasn't what they were looking for, the user would click another video, and if needed yet another one.

All these clicks were interpreted as appreciation for the content, when on the contrary they were a sign of frustration. For this reason, the criterion was later changed to also include watch time.

And that's not all: a user can watch an entire video without enjoying it, perhaps because they're hoping to find exactly what they were looking for. For this reason, the metric was later refined further to also include the rating given to the content.

The effect of changing the reference metric was an immediate 20% drop in clicks. But this seemingly counterproductive result is actually a sign of appreciation from users.

Why you shouldn't chase the North Star Metric

But if the North Star Metric measures long-term success, why do our clients come to us with a misleading signal coming from this very metric?

The reason is explained by a British economist, Charles Goodhart, who stated in 1975 that when you try to control a measure, it becomes distorted. In other words, what would come to be known as Goodhart's law holds that trying to improve an indicator ends up making it an invalid measure.

A few years later, the American social psychologist Donald Campbell backed him up: in a study from 1979, he made a statement that he himself would describe as pessimistic:

What does this statement, which would come to be known as Campbell's law, tell us? Put simply, that if a metric drives decisions, it is likely to be manipulated to the point of damaging the very measure it was supposed to increase.

How controlling the metric harms the app

But neither Goodhart nor Campbell were talking about digital products; so how is it possible that their statements also apply to the context we're interested in? And how can a good metric become invalid precisely when it seems to be improving?

Let's take an example. In a language-learning app, a possible North Star Metric could be the number of minutes spent studying. To increase the metric, however, an action is taken such as increasing the number of minutes in each lesson; this way, every user who completes even a single lesson will take ten minutes instead of five.

The result, however, is that users who don't have ten minutes to spare won't bother completing any lesson at all. As a result, there are fewer and fewer active users, the number of lessons completed per user per day drops and, overall, appreciation for the app declines.

And that's not all: sometimes, to achieve an improvement in metrics, people resort to underhanded methods. For example:

• To increase the number of sign-ups to the app, you can hide the button to continue without a registered account.
• To get more clicks on your blog, you can choose clickbait titles.

Adopting these patterns can lead users to distrust the app, achieving the opposite effect of the one you hoped for.

How to use the North Star Metric correctly

Trying to alter metrics directly is like putting the thermometer in ice when you have a fever: sure, the temperature will drop, but that doesn't mean the flu is gone.

On the flip side, this doesn't mean you can just sit back and wait. The North Star Metric works exactly like a thermometer: knowing that your body temperature is high tells you something is wrong, and that's valuable information.

This way, we can act on the causes we assume to be behind the malaise: an improved onboarding, a missing feature, an unclear flow. If the overall health of the product improves, our key metric will improve too.

A practical guide

So we've established that data should be used to understand the big picture of the situation, not to alter it. But how can we act so as not to fall into Campbell's trap?

First of all, let's choose the right metric. The North Star Metric should measure value received by the user, not just a quantity. For example, rather than sign-ups to the digital product, it might make sense to measure how many users perform the main action the app is designed for within a week of signing up.

Then, the work must be aimed at acting on input metrics, that is, hyper-specific criteria that can be influenced through day-to-day work, such as the number of features released in a quarter. They are the counterpart of output metrics, the name given to all those metrics that represent a result and cannot be directly controlled.

Every metric considered as an achievable result must be balanced by a counter metric, whose purpose is to indicate whether the metric is being improved in the right way. For example:

• If you're looking at the number of sign-ups, you should also keep track of retention after one month.
• If you increase the number of notifications sent, there shouldn't be a sharp drop in the number of users who enable notifications.
• If you aim to make users spend more time in the app, satisfaction and positive reviews shouldn't drop.

Finally, try to anticipate possible loopholes: ask yourself whether, and how, you could increase your North Star Metric without improving the product. This way, you'll know exactly what to avoid doing so that number doesn't lose its value.

The North Star Metric is currently one of the most useful tools for giving your digital product the right direction. The problem arises when it stops being a compass and becomes a finish line.

Has this ever happened to you? If you'd like an outside perspective to define the right metrics for your digital product, or to understand what your data is really telling you, get in touch: we'd be happy to talk about it.