The 5 steps of optimizing around failure
The key difference between people who analyze data and people who don’t is that people who analyze data know that they’re usually wrong, and people who don’t analyze data incorrectly assume that they’re usually right. Ideally analyzing data makes us humbler, more realistic, and more tolerant about others’ mistakes. It can also help us get better at being right next time. It’s easy to tell yourself the opposite, to look down on the number crunchers and say “ah yes, but unlike you, I’m a visionary, I don’t need no stinkin’ numbers .” Good luck with that. Many of the number crunchers are also visionaries, they’re just visionaries with a better respect for where things go wrong and with a better chance of getting things right. Our nature is to fail. Many find that view depressing. I find it invigorating. You need to build your strategies around these realities. And once you realize that failure is natural and usually inevitable, you can stop feeling defeated by it. You’ll also better appreciate your successes. Google’s Udi Manber recently told Business Week: “We ran over 5,000 experiments last year. Probably 10 experiments for every successful launch.” The 5 steps of optimizing around failure:
Embrace its inevitability. Don’t put all your efforts into trying to prevent it.
Fail early and often. And minimize the downside risk. Realize that each effort has a reasonable chance of failing, and don’t bet the company on it. Don’t spend man-months on design, development, and testing.
Have a clear and important goal, along with a falsifiable hypothesis, and a way of measuring against it. Such as “this change should increase RPM by at least 5%” or “this should decrease spam by 10%.”
Have the discipline to measure honestly and to revert appropriately. One of my favorite compliments was when a colleague said “I’m always happy to run tests for Gil, because I know that if it fails he’ll be the first to admit it.”
Get better. Figure out what went wrong and do better next time.
The zeroth rule, the premise behind the whole endeavor, is that you must first recognize that you’re usually wrong. Until you can do that, your chances of success are slim. Once you embrace the frequency of your fallibility you can greatly increase your chances of success, as well as your appreciation of it.
