What is survivorship bias?

Survivorship bias is a selection bias where your data has already been filtered by survival, by making it to the point where you can observe it. You judge a process from the survivors (the visible numerator) while the eliminated cases (the rest of the population) are truncated and invisible, so conclusions skew optimistic. The classic example: WWII planes that returned showed damage on the wings and fuselage, but the planes hit in the engine never came back to be counted, so the clean spots on the survivors were actually the fatal ones.

What did Abraham Wald actually do?

At the Statistical Research Group at Columbia, Wald wrote eight 1943 memoranda titled 'A Method of Estimating Plane Vulnerability Based on Damage of Survivors.' Using only the returning planes and where they were hit, he derived equations to reconstruct the damage distribution of the planes that did not return. He did not discard the survivors; he reasoned from them while correcting for the selection filter that produced them.

Is the famous Wald plane story true?

The logic is real, but much of the popular story is a modern reconstruction. Wald wrote about 'a plane' generically, not the B-17. He never wrote 'armor the engines'; that clean line is a later distillation by writers like Jordan Ellenberg, not a Wald quote. The dramatic scene of Wald confronting the generals is undocumented, and the iconic red-dot bomber diagram was created by designer Cameron Moll around 2005, not during the war. Only the tidy version survived, which is itself a small case of survivorship bias.

Heuristic ▶ Subscribe

Episode 4 · Cognitive biases · Research notes

Armor the bullet holes? The trap in the most famous statistics story

WWII planes came home riddled with bullet holes, and the obvious fix was to armor where the damage was. A statistician named Abraham Wald said the opposite: armor the clean spots. The planes hit there never came back to be counted. That gap between the winners you see and the losers you don't is survivorship bias.

The short answer: the damage on the survivors shows where a plane can be hit and still fly, so the untouched areas are the real kill zones. The trap is judging any process only by its survivors. And the honest twist most videos skip: the survivorship logic is brilliant and true, but the famous version of this story is largely a modern reconstruction, right down to a red-dot diagram made around 2005.

Episode 4 premieres

Wednesday, July 22 · 8:00 AM PT

▶ Subscribe to catch it

What Wald actually did

Abraham Wald fled the Nazis in 1938; most of his family was later killed at Auschwitz. In America he joined the Statistical Research Group at Columbia, a secret team of elite statisticians. His 1943 memoranda, "A Method of Estimating Plane Vulnerability Based on Damage of Survivors," used only the returning planes to reconstruct the damage the missing planes must have taken. The skill was not refusing to look at winners; it was modeling the filter that produced them.

Where the story is half legend

The logic is real. Most of the drama around it is not. Wald wrote about "a plane," generically; the heroic B-17 framing came later. He never wrote "armor the engines," that memorable line is a distillation by later authors (notably Jordan Ellenberg), not a quote. The scene of Wald facing down stubborn generals is undocumented. And the iconic diagram of a plane peppered with red dots was made by designer Cameron Moll around 2005, his own dots on a modern aircraft outline. Over the decades, only the tidy version survived, which is itself a quiet case of survivorship bias.

Where it fools you

Mutual funds that fail get shut down and quietly drop out of the long-run averages, so the surviving-fund average overstates real returns; correcting for the dead funds can pull a headline figure down by a percentage point or more. "This billionaire dropped out of college" shows you Gates and Jobs, not the vast invisible crowd who dropped out and did not win. "They don't build them like they used to" compares today's average to history's survivors, after the flimsy majority already rotted away. In each case the missing denominator, the failures nobody kept, is exactly the data you never see.

Is it always a trap?

No, and that is the honest part. Wald looked at winners; survivor data can be perfectly valid. The error is only ignoring the filter, and the danger scales with how tightly survival is tied to the thing you are measuring. Some famous examples are shakier than they sound, the claim that cats fall from higher floors and get hurt less is a real study but a contested survivorship reading, with a later study finding the opposite. And in controlled experiments (Enke, 2020), many people do ignore hidden data, but a single reminder to "consider what you are not being shown" roughly halved the mistake. It is a default that slips under load, not a fixed flaw.

Sources

Wald, A. (1943). A Method of Estimating Plane Vulnerability Based on Damage of Survivors. Statistical Research Group, Columbia (CNA reprint CRC 432, 1980).
Mangel, M. and Samaniego, F. J. (1984). Abraham Wald's work on aircraft survivability. JASA, 79(386), 259-267.
Casselman, B. (2016). The Legend of Abraham Wald. AMS Feature Column.
Elton, E. J., Gruber, M. J. and Blake, C. R. (1996). Survivorship bias and mutual fund performance. Review of Financial Studies, 9(4), 1097-1120.
Denrell, J. (2003). Vicarious learning, undersampling of failure, and the myths of management. Organization Science, 14(3), 227-243.
Tversky, A. and Kahneman, D. (1973). Availability. Cognitive Psychology, 5(2), 207-232.
Enke, B. (2020). What you see is all there is. Quarterly Journal of Economics, 135(3), 1363-1398.
Whitney, W. O. and Mehlhaff, C. J. (1987). High-rise syndrome in cats. JAVMA, 191(11), 1399-1403.