Numbers lie
Bruno Martinaud
Entrepreneurship Academic Director, Ecole Polytechnique
It’s 2009. Kevin Systrom (soon joined by his co-founder, Mike Krieger) is working on a geolocation social media project, similar to Foursquare. Together, they manage to convince Baseline Ventures and Andressen Horowitz to invest $500,000 in the project. This enables them to dedicate themselves full time to the adventure. A year later, Burbn is launched in the form of an iPhone application that makes it possible to save locations, plan outings, post photos, etc. The application is downloaded massively, but the verdict is not quite what they hope for: the users, beta-testers, don’t like it at all. Too cluttered, too messy, it’s confusing and most of them have stopped using it. A patent failure. All this being very normal, the entrepreneur digests the feedback, learns from the experience and moves on to a new adventure. The metrics are bad – duly noted. And yet Kevin Systrom doesn’t stop there because he notices something that at first glance seems trivial: the photo sharing function (one amongst so many others) seems to be used by a small number of regular users… He investigates, questions these users and realises that the small group loves this function (and only this one). Instagram is born, all from the happy realisation that a small number of people, hidden in the multitudes that didn’t like Burbn, use the app for one reason.
This story highlights a counter-intuitive principle for the educated manager: numbers lie in the beginning. Burbn’s metrics were catastrophic. The rational response would have been to acknowledge that fact and move on to the next project. But a weak signal was hiding there, showing potential.
The story of Viagra follows a similar pattern. Pfizer laboratories were developing a blood pressure regulator, which was in phase III of testing before gaining market authorisation. If we remember that the development of a new molecule represents an investment of approximately $1bn, that would mean around $700m to $800m had already been invested in the project. Pressure was therefore high to achieve this authorisation as soon as possible. It just so happened that someone in Pfizer’s teams noticed that some people in the test sample hadn’t returned the pills that should have been left over as part of the procedure given to them. Who pays attention to that? Some incoherent data, with no direct link to the topic (efficiency of the molecule)… A few abnormal results in a table of 300 columns and 100,000 rows… And yet, by investigating, this person realised that those who weren’t giving back the extra pills all shared the same characteristics of age and sex. Pfizer then realised that this blood pressure regulator had an unexpected side effect so interesting that the project changed course entirely.
A simple observation lies behind these examples that we can compare endlessly an innovative project, a start-up that is just starting up, they are adventures to be explored.
Exploring first means remembering that you don’t know what works and what doesn’t work in your idea. It’s recognising that you’re facing complex issues, that you don’t quite grasp all the variables of these issues and don’t understand how the variables interact, or their effects.
From that starting point comes the following consequence, the subject of this article: you don’t know what to measure and you don’t know the meaning of what you’re measuring. This goes both ways: what might initially seem like poor metrics, as in the case of Burbn, can hide a gem. But the opposite is also true. We have recently worked with a start-up developing a smart object for well-being, aimed at the public at large. The company quickly sold some tens of thousands of the product, and based on this success, raised funds to scale up quickly and control the market, only to find that its sales, far from growing, plateaued and then fell. It turns out that 30,000 products being sold wasn’t the sign of massive and rapid market adoption, but the majority of the addressable market. After a period of trying different things, questioning themselves, doubting and researching, the start-up’s founders finally found a B2B market, centred on a service offer based on the smart object. The irony is that its strong early figures didn’t mean that it had found its market.
These observations lead to two simple and practical recommendations, which seem almost trivial when writing them, but they can be slippery in their application:
1. Remember that the only way to progress in a complex environment is through experimentation. Trial and error. Keep what works. Eliminate what doesn’t. Understanding will come later. Pixar has always applied this empirical approach to the extreme. From a starting concept, Pixar tests everything. There have been, throughout the production process, 43,536 variations of Nemo, 69,562 of Ratatouille and 98,173 of Wall-E… That’s the path between initial idea and final success.
2. Give yourself the tools to ‘capture’ weak signals, that is, put strategies in place to save what seems irrelevant in one instant but which could be useful later. Remember that at a given moment, in the first life of an innovative project, no one is able to determine what is relevant and what is not.
Unfortunately, the human mind is wired in such a way as to try to give early meaning to the information that comes to it, which leads to neglecting the need to test everything (because we’ve already understood) and to filtering out noise (because we’ve already identified the signal)… These are probably the two deadly sins of the innovator or the start-up entrepreneur.