# Abraham Wald survivorship bias intuition

During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire. Wald noted that the study only considered the aircraft that had survived their missions; any bombers which had been shot down or otherwise lost had logically also been rendered unavailable for assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. So Wald proposed that the Navy reinforce areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost.

So here's the thing that comes to mind: From what intuition did Wald draw such an idea? Because as a statistician, I assume that also for the rigor of his work, he very probably had to demonstrate through mathematical proofs that in fact it was necessary to reinforce the armour exactly where the holes are not, and not simply force the armour first and then observe if that would resulte positive.

Was his idea born from some formula? From what pattern of thought could he do just the opposite of what all his statistical colleagues had thought? For one thing to me to have this insight intuitively, and for another thing it is to be able to describe it mathematically.

• Is that a historical image or just an illustration? Commented May 8, 2020 at 15:17
• Can you please give the source of the image, because it is a wonderful illustration that I would love to use in my future teaching on this subject.
– Ben
Commented May 10, 2020 at 6:53
• @Ben-ReinstateMonica, the image appears to come from the Wikipedia page. The image caption says, "Image shows hypothetical data" (alas). Commented May 13, 2020 at 18:41

Wald wrote his analysis in a set of eight technical memoranda for the Statistical Research Group (SRG), and these documents have been published by the Centre for Naval Analysis, so you can actually read them and see exactly what he did in his written work. The original memos can be found here and a good summary and analysis of his work on survival of aircraft is set out in Mangel and Samaniego (1984). If you read this paper and the accompanying technical memos you will see that Wald did indeed develop a theoretical model of aircraft survival to make his conclusions.

Wald's analysis covered multiple problems, starting with estimation of the survival probabilities for aircraft based on the number of hits to those aircraft, and then proceeding in more detail to estimation based on the number and location of the hits. His method goes essentially like this. For any plane that leaves on a bombing run, there is a distribution for the number of hits that plane will take. For any given number of hits there is an associated survival probability; a plane with no hits is assumed to survive with probability one, and the conditionl probability of survivial from each hit is some positive probability. Based on this, it is possible to use data on the returning planes to estimate the conditional probabilities of survival for each hit. (Any plane that does not return is assumed to have been hit at least once, and is missing data.)

To deal with different probabilities for hits to different locations, Wald expanded his model to divide the plane into a finite number of locations, and expanded the model so that there is an unknown conditional probability of survival for each hit to each location. Wald assumed that the conditional distribution of hits to specific locations (given the total number of hits) was fixed. He advanced the view that this could be determined experimentally within the military research centres by firing dummy bullets at real aircraft. Given assumed values for these probabilties (based on independent experiments), and some sensible statistical independence assumptions, you can then use the data on the number and locations of hits on returning planes to estimate the conditional probabilities of survival for each hit on a particular location. (Again, any plane that does not return is assumed to have been hit at least once, and is missing data.) If you formulate a model like this, the phenomenon under consideration here will emerge --- i.e., the more hits observed on a particular location, the higher the estimated survival probability for hits on that location will be. Thus, the parts of the aircraft showing the lowest survivial probability for hits will be those that have the lowest number of hits in the returning planes (relative to the probability of being hit in that location).

Wald's technical memoranda on this subject give specific formulae for his models and estimation methods for the unknown conditional probabilities. Nevertheless, it is not necessary to understand the technical details to understand his basic insight that areas of the plane exhibiting low numbers of hits (on the returning planes) are actually the most vulnerable parts. The reason this story is so wonderful is that it illustrates a counter-intuitive problem involving missing data where the intuitive answer for most people is the opposite of the correct answer. I have used this problem as a teaching example in statistics seminars a number of times, and when asked their advice, the audience inevitably advise that more armour should be attached to the parts of the planes that showed the most damage in the data. When they are informed of Wald's opposite advice, and the reasoning for it, they are usually very impressed and realise their error.

Finally, with regard to the actual folk legend that has emerged around this, I am not aware of any evidence showing exactly what conversations Wald had with the military, what he said, or who disagreed with him (or was surprised by his advice). However, in the versions of the story I have heard, it was not other statisticians he was disagreeing with, but military officials who lacked the statistical training needed to have the intuitively correct answer to this missing data problem. In any case, it is a brilliant story about a brilliant man.

• Do you know of any R or other statistical packages that can be used to apply his methods? E.g. picture you take the titanic dataset -- get rid of all the attribute information associated with those who died. How would Wald build a model that predicted surviving? (Is there a software package that implements this...?) Commented May 20, 2021 at 16:25
• @BryanShalloway: Sorry for the long delay in responding; I just noticed this comment. I'm not aware of any R package specifically for this, but it would essentially be a kind of hierarchical survival model.
– Ben
Commented Jan 29, 2023 at 0:25

I thank all those who have had their time bringing an answer.

In the meantime, I have been looking for more on the subject and found his article with the intuitions and mathematical proofs. Follow this link:

https://apps.dtic.mil/dtic/tr/fulltext/u2/a091073.pdf

Really a remarkable statistician!

• Nice work finding the memos (+1). I have updated my own answer to add the link.
– Ben
Commented May 14, 2020 at 0:40

I think it is just that Abraham Wald was a clever man and an acomplished statistician, and saw that aircraft observed were not a random sample, but rather a sample self-selected by survivorship.

I do not think that a "mathematical proof" can be given that certain concrete areas need reinforcing. Rather what Wald did is a sensible descriptive analysis of the data at hand, supplemented with sound statistical judgement. I imagine that when he proposed to reinforce areas with no impacts, subject-matter specialists looked upon those areas and confirmed that they were the critical ones to protect.

• On absence being informative as well as presence, see e.g. bigthink.com/artful-choice/… Commented May 8, 2020 at 9:17
• This answer does not appear correct to me. Wald's memoranda go far beyond descriptive analysis; they include models for estimating the conditional probabilities of survival based on hits to areas of the plane. His models seem to me to constitute "proof" of the phenomenon (so far as statistical models ever do this.). All Wald's memos on this are published, so there is no need to speculate on what we imagine his reasoning might have been.
– Ben
Commented May 10, 2020 at 3:03
• I have not read the memoranda you mention: I know this story second hand, from one of his students (who was in turn one my professors). If you have read said memoranda and they say what you say, I gladly accept your correction. I did not say that Wald's work was only descriptive, but rather that built on clever description. Let me point also that the question asked about what may have led Wald to reason along the lines he reasoned ("From what intuition did Wald draw such idea?") on which I think it is fair to speculate. Commented May 10, 2020 at 10:39
• An additional (and most important) point is that I think no "mathematical proof" can be given that a certain area needs reinforcing. One may offer evidence that damage in certain parts appears to make survival difficult of impossible, much as you can find evidence that a certain drug protects from coronavirus. This is no "mathematical proof" that you have to take said drug (maybe another one would be preferable). It is only evidence for the subject-matter specialist to reflect and decide. Commented May 10, 2020 at 10:44
• I think you can still show rigorously that the probability of be downed if you are hit in certain places is higher than in other. Thus you should reinforce those places. Commented Oct 9, 2023 at 14:18