The Power of the Unseen

During World War II, the American military enlisted the help of the Statistical Research Group (SRG) — a consortium of the best American statisticians in the world at the time — to analyze wartime data and make recommendations for how best to armor their planes. Since armor protects planes from gunfire but simultaneously weighs them down, hindering both their maneuverability and fuel-efficiency, getting the answer right meant a better chance of winning the war. And there was one statistician in particular, an Austrian by the name of Abraham Wald,¹ whose statistical brilliance was unrivaled. So when Wald offered his opinion on where best to place armor on the planes, people listened.

Upon examining the planes that returned from war, Wald noted that the bullet holes were most prevalent among the wings, tail, nose, and fuselage. The typical conclusion to make, then, was that the armor should go where the planes were getting hit. But that conclusion missed what Wald didn’t: the sample from which the data was being pulled was a sample that included only the planes that made it back from war. It wasn’t true, after all, that planes were only getting hit in the wings, tail, nose, and fuselage; it was that those hits didn’t bring down the planes — being hit in the engine did, and the planes that got hit in the engine didn’t make it back. Wald’s insight, then, was to point out what would at first seem counterintuitive: the armor should go on the engines, where the bullet holes aren’t.

In his book, How Not to Be Wrong: The Power of Mathematical Thinking, mathematician Jordan Ellenberg illustrates the brilliance of this insight in the context of survivorship bias, defined as “the logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility.” Wald’s insight was a perfect example of overcoming this bias, as he understood that in order to optimize the armoring of the planes, it was necessary to focus both on the planes that returned from war and on the planes that didn’t; the former to understand where on the planes didn’t require armor, and the latter to understand where on the planes necessarily did.

As Ellenberg points out in his book:

“One thing the American defense establishment has traditionally understood very well is that countries don’t win wars just by being braver than the other side, or freer, or slightly preferred by God. The winners are usually the guys who get 5% fewer of their planes shot down, or use 5% less fuel, or get 5% more nutrition into their infantry at 95% of the cost. That’s not the stuff war movies are made of, but it’s the stuff wars are made of. And there’s math every step of the way.”

Survivorship bias exists beyond wartime strategy, however. The world we live in, after all, is a result of what has survived. To ignore what hasn’t is to miss the miraculousness of what has; for every business that has succeeded there are countless more that have failed (for every Best Buy, a Circuit City); for every menu item there are hundreds that never got ordered, and were eventually removed (for every Big Mac, a Big ‘N Tasty); for every species there are millions that were not evolutionarily fit enough to survive up to the current era (for every human, a Dodo).

To overcome the survivorship bias that we all experience is to see the fullness of the world by understanding just how unlikely the survival and subsequent existence of everything in it is. Beyond that, however, it is to understand the impossibility of arriving at a complete understanding of the world by analyzing only that which exists currently, just as an armoring strategy based only on analyzing the planes that returned from war would have led the Allies astray in their quest to win the war. Evolutionarily, seeing past what existed served no purpose. But in a world as complex as ours is, the advantage of doing so — of overcoming survivorship bias — is seeing what does exist as a result of what has worked, and more importantly, what doesn’t as a result of what hasn’t. One place this insight — and resulting strategy — is particularly relevant is in the context of small, local businesses.

If you’re a business owner — let’s say, of an especially good restaurant— you tend to get a lot of repeat patrons. These are the people who enjoyed your food enough to come back, again and again. (In the context of survivorship bias, they are those “who made it past some selection process,” or the planes that made it back.) What is as important to consider if you’re a restaurant is the patrons that came in once but did not return, or in the context of Wald’s reasoning, the planes that didn’t make it back. They are the patrons you do not see again, so they are easy to ignore, but to ignore their feedback is to make the mistake Wald wisely avoided: the development of a strategy based on an incomplete sample. Thus, if you are a restaurant looking to attract new patrons, you must focus on two things: first, that which is currently working, for which the feedback you receive from repeat patrons is an effective proxy, and second, that which is not. The latter, of course, is where reviews come in handy, especially negative reviews; they are the pieces of feedback which, per survivorship bias, you might never otherwise have been exposed to.

“How was everything?”

“It was great, thank you.”

*waiter leaves*

*honest feedback commences between diners*

Of course, reviews are an imperfect method of doing this; review sites are constantly combatting efforts by bad actors to game the system, whether by soliciting positive reviews for their own businesses or writing fake negative reviews for competitors. But while it is true that review sites are imperfect, both in terms of moderation and in terms of how the feedback is delivered, it is clear that their content can prove invaluable to businesses, which, by nature of survivorship bias, might never otherwise get insight into what exactly drove the patrons who never returned to never return. In the same way that Wald utilized data from an otherwise unseen sample to craft an optimal armoring strategy for the Allies’ military in World War II, business owners can utilize reviews to strategize based not just on what is already working, but on what they might otherwise have no idea is not. This is the power of the unseen.