The Problem with Vanity Metrics

mojeek

27 September 2013

2 min

Team

Vanity metrics...

Like the proverbial chocolate teapot, you can infatuate over it with your friends, perhaps even an investor (if it's an especially large incarnation). But that's worth diddly-squat when you demonstrate its unique selling point. It's just a big, hollow disappointment.

So why do vanity metrics even exist?

The startup world is swimming in data - whether it's the premise of your company or the 17th pitch deck you're presenting to investors. So perhaps it shouldn't come as any surprise that 'vanity' elbows its way into the equation.

We're incentivised to present our products/ideas/companies in the best possible light, and as humans, we have the remarkable ability to delude ourselves into lying - somewhat innocently. In psychology this is called the egocentric bias [1] (or in extreme cases, the Dunning-Kruger effect [2]).

So what can be done about this?

Well, here's the problem. Self-delusion is not entirely conscious. If it was, it would be a moral problem - which is a very grey area. People would be fully aware that they are over-arching the reality of their idea, and could (with a lot of effort) change their attitudes.

Fortunately, we take a diversion from that messy locale and find that it is an environmental problem, which is easier to manage. Let me explain\...

If behaviour is not being directed from within, it must be directed from outside - specifically, the environment. If you place a lab rat into a small maze where it can only turn left, its behaviour is going to reflect that - clearly. (Unless it's watched Men Who Stare at Goats of course).

So what is a solution?

To bring this problem some respite, a possible solution would be to further qualify every metric that you list and to try your best to remain cognizant of these psychological limitations that we all have. Indeed, part of the reason why these irrational behaviours are so common is because people assume that they see the world as it really is - so-called naive realism [3]. So I would venture that it's a combination of accountability and education.

Whatever it is, it's about time we ate the chocolate teapots, and focused on ways to make a truly great cup of tea.

Some examples of vanity metrics

Don't tell investors that you're driving 20,000 uniques to your site when 50% are bots.
You've got an email list of 40,000 - who cares? What's your conversion rate?
You've grown by 600% - but have you grown by 6 or 600,000 users?
You've sold £100,000 worth of product over the last 6 months - what's your margin?

You know the websites that are driving the most traffic - but do you know the websites that are driving the most conversions?

As serial entrepreneur Eric Ries said in this great post on vanity metrics, measure what matters