Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Original question (7/25/14): Does this quotation from the news media make sense, or is there a better statistical way of viewing the spate of recent plane accidents?

However, Barnett also draws attention to the theory of Poisson distribution, which implies that short intervals between crashes are actually more probable than long ones.

"Suppose that there is an average of one fatal accident per year, meaning that the chance of a crash on any given day is one in 365," says Barnett. "If there is a crash on 1 August, the chance that the next crash occurs one day later on 2 August is 1/365. But the chance the next crash is on 3 August is (364/365) x (1/365), because the next crash occurs on 3 August only if there is no crash on 2 August."

"It seems counterintuitive, but the conclusion follows relentlessly from the laws of probability," Barnett says.

Source: http://www.bbc.com/news/magazine-28481060

Clarification (7/27/14): What is counter intuitive (to me) is saying that rare events tend to occur close in time. Intuitively, I would think that rare events would not occur close in time. Can anyone point me to a theoretical or empirical expected distribution of the time between events under the assumptions of a Poisson distribution? (That is, a histogram where the y-axis is frequency or probability and the x-axis is time between 2 consecutive occurrences grouped into days, weeks, months, or years, or the like.) Thanks.

Clarification (7/28/14): The headline implies it is more likely to have clusters of accidents than widely spaced accidents. Lets operationalize that. Let's say that a cluster is 3 airplane accidents, and a short period of time is 3 months and a long period of time is 3 years. It seems illogical to think that there is a higher probability that 3 accidents will occur within a period of 3 months than within a period of 3 years. Even if we take the first accident as a given, it is illogical to think that 2 more accidents will occur within the next 3 months as compared to within the next 3 years. If that is true, then the news media headline is misleading and incorrect. Am I missing something?

share|improve this question
1  
Re the clarification: You might find it helpful to distinguish between probability, probability per unit time, and expectation. Although processes describing rare events will--practically by the very meaning of "rare"--have a long expected time between events, that is not inconsistent with the probability per unit time being greatest at the outset. Nevertheless, the probability of the event next occurring within a short time will be very small. –  whuber Jul 27 at 17:35
2  
Also, I just noticed this Wikipedia article--you might like it. Oh, and I just came across this pdf, too--it specifically mentions the "clustering" of airplane crashes (and describes the issue much better than I have so far...). –  Steve S Jul 28 at 19:07
1  
@Glen_b: The flaw in the newspaper article (implied in the title of the article, which is the title of my posting) is that the article suggests there is a higher probability of a given number (i.e., a cluster) of accidents occurring in a short period of time than over a longer period of time. That is just wrong. –  Joel W. Jul 29 at 9:59
1  
@JoelW.: If anything, it would be the journalist that screwed up... Anyway, is everything cleared up or do you still have any reservations left over? –  Steve S Jul 30 at 20:55
1  
My guess is that it was the statistician who misled the journalist. I doubt the journalist got it wrong on his/her own (because it is so counter-intuitive). –  Joel W. Jul 30 at 23:50

4 Answers 4

up vote 2 down vote accepted

Summary: The first sentence in the quoted BBC paragraph is sloppy and misleading.

Even though previous answers and comments provided an excellent discussion already, I feel that the main question has not been answered satisfactorily.

So let us assume that a probability of a plane crash on any given day is $p=1/365$ and that the crashes are independent from each other. Let us further assume that one plane crashed on January 1st. When would the next plane crash?

Well, let us do a simple simulation: for each day for the next three years I will randomly decide if another plane crashed with probability $p$ and note the day of the next crash; I will repeat this procedure $100\,000$ times. Here is the resulting histogram:

Distribution of plane crushes, a model

In fact, the probability distribution is simply given by $\mathrm{Pr}(t) = (1-p)^t p$, where $t$ is the number of days. I plotted this theoretical distribution as a red line, and you can see that it fits well to the Monte Carlo histogram. Remark: if time were discretized in smaller and smaller bins, this distributions would converge to an exponential one; but it does not really matter for this discussion.

As many people have already remarked here, it is a decreasing curve. This means that the probability that the next plane crashes on the next day, January 2nd, is higher than the probability that the next plane will crash on any other given day, e.g. on January 2nd next year (the difference is almost three-fold: $0.27\%$ and $0.10\%$).

However, if you ask what is the probability that the next plane crashes in the next three days, the answer is $0.8\%$, but if you ask what is the probability that it will crash after three days, but in the next three years, then the answer is $94\%$. So, obviously, it is more likely that it will crash in the next three years (but after the first three days) than in the next three days. The confusion arises because when you say "clustered events" you refer to a very small initial chunk of the distribution, but when you say "widely spaced" events you refer to a large chunk of it. That is why even with a monotonically decreasing probability distribution it is surely possible that "clusters" (e.g. two plane crashes in three days) are very unlikely.

Here is another histogram to really get this point across. It is simply a sum of the previous histogram over several non-intersecting time periods:

Histogram of plane crushes frequency

share|improve this answer
    
Are you saying that the MIT professor is wrong? –  Steve S Aug 18 at 23:48
    
No, the quote from Barnett in the BBC article is completely correct. But its interpretation by the BBC reporter is sloppy at best: "Barnett also draws attention to the theory of Poisson distribution, which implies that short intervals between crashes are actually more probable than long ones". The most natural interpretation of this sentence is dead wrong (and I suppose Barnett did not mean to imply that). Maybe I should be more explicit about that in my reply. Is there any substantial part of my answer with which you disagree? Hope not, as I fully agree with yours. –  amoeba Aug 19 at 8:35
    
@SteveS: I updated my "summary", hope it will be less confusing now. –  amoeba Aug 19 at 11:07
    
I'm sorry if my initial comment came across as curt. –  Steve S Aug 19 at 21:30

What the reporter is saying is that the random occurrence of a plane crash can be modelled as a Poisson process--a situation where the probability of an event occurring over some (small) interval is proportional to the length of said interval and where each occurrence in Independent of all others.

Is this a reasonable model for the scenario described?

Probably.

Sure, these events might not be 100% Independent since other pilots likely alter their behavior (if only very slightly) after a crash. [I don't know--perhaps a few pilots do some extra bit of simulator training or something like that]. Nevertheless, the assumption of Independence is still entirely reasonable.

What about clusters of plane crashes?

Yes. Given a Poisson process (or even some other random process), you would expect to see some clusters of occurrences.

In fact, as described by the Oxford Dictionary of Statistics in its entry for Poisson Process (which is a "mathematical description of randomness"):

[R]andomness usually gives rise to apparent clustering, despite the natural
expectation that randomness would lead to regularity.

For example, check out this simple bit of R code:

set.seed(123)
x <- runif(500)
y <- runif(500)

plot(x, y, pch=20, col='blue', main="A Random Distribution of Points")

which produces:
Notice the clumping?

Even though we know this is a plot of random points, it sort of looks like there are some non-random bits to it--specifically, in some parts of the graph there are clumps of points while other parts are wide open. It's this same sort of behavior that the article is trying to describe (only with time series data and not spatial data).


UPDATE:

@JoelW.: So, for instance, let's say the probability of a plane crashing tomorrow (or any day for that matter) is "p" (and, let's say "p" is something like 1 in a hundred).

The reason why the next plane crash is more likely to occur tomorrow than it is more likely to occur in exactly a year (i.e. on July 26, 2015) is because the probability that the next crash is in exactly one year is equal to:

= Prob(crash tomorrow) * Prob(365 days with *no* crashes)

Make sense?

Ultimately, I think that the reason these things are Counter-Intuitive is because usually when we think of a phrase like: "The odds of a plane crash in one month compared with the odds of one happening tomorrow". We naturally don't immediately consider the 24-hour period that begins in exactly one month. Instead, we (or at least I do) tend to think of it in more, well, flexibly. So more like: a month ± a week. That and the fact that we forget about taking into account the odds of a crash not happening in the interim... (But again, maybe that's just me...).

Phew!


Additional Resources:

  • Wikipedia's article on the Clustering Illusion
  • A pdf which specifically mentions the "clustering" of plane crashes (on page 8) and briefly describes the mathematics of a Poisson process.
share|improve this answer
1  
@Joel W.: Actually, I should add more to this answer--give me a couple minutes to edit... –  Steve S Jul 25 at 20:56
7  
The argument for delaying travel is the same one appearing in the old joke about how the TSA found a statistician with a bomb on board an airplane. When asked to explain herself, the statistician said "Well, the odds of one person having a bomb are small yet not small enough for comfort, but the odds of two people having a bomb are infinitesimal. Therefore when I bring a bomb, there is almost no chance there will be two bombs and we will be perfectly safe." –  whuber Jul 25 at 22:25
1  
Your joke is on point, @whuber, but there seems to be some sort of logical disconnect between saying that "short intervals between crashes are actually more probable than long ones" and saying that the probability of a crash tomorrow is independent of whether a crash occurred today. I guess probabilty can be counter-intuitive. –  Joel W. Jul 25 at 23:02
1  
What is counter intuitive (to me) is saying that rare events tend to occur close in time. Intuitively, I would think that rare events would not occur close in time. Am I the only one with that intuitive view? –  Joel W. Jul 27 at 2:21
1  
@Steve S: Thank you for the link. What would the exponential distribution look like for the assumed value in the news article (1/365)? In any case, perhaps the Exponential Distribution does not address the headline of the article, which implies a comparison of the probability of a given number of events happening within a short time period with the probability of that number of events happening within a long time period. –  Joel W. Jul 28 at 13:00

If the number of plane crashes is Poisson distributed (as he seems to be stating), the time between crashes has an exponential distribution. The pdf of the exponential distribution is a monotone decreasing function of time. Hence earlier crashes are more likely than later crashes.

share|improve this answer
    
"short intervals between crashes are actually more probable than long ones" How is this different from saying that if there has just been a plane crash we should all delay our upcoming travel (for statistical reasons)? –  Joel W. Jul 25 at 22:19
2  
Joel, That quotation is meaningless until its author quantifies what is meant by "short" and "long". In his example of an event with an expected rate of one per year, the chance of a recurrence during the next month will still be far less than the chance that the next crash occurs more than one year later. What he might have meant is that the probability per unit time is greater in the near term than in the longer term. To compare actual probabilities you have to multiply the probability per unit time by the duration (technically, you have to integrate it over the duration). –  whuber Jul 27 at 17:39
    
@whuber: The headline speaks of the probability of a cluster of airplane accidents. Nothing said on stackexchange so far has convinced me that a cluster of airplane accidents is more commonplace or likely than widely spaced airplane accidents. So, it seems to me that the quotation from the news media is downright misleading (perhaps because the time intervals are not identified, as you wrote). What do you think? –  Joel W. Jul 27 at 19:10
    
I don't know what you mean by "widely spaced airplane accidents" nor, for that matter, am I completely sure what you understand a "cluster" to be. Suppose, to make the situation concrete, a series of rare events occurs in years 0, 10, 11, 12, and 22 (counting from some initial date). Exactly how many "widely spaced" events have occurred? How many "clusters" have occurred? I can find defensible answers to the first question ranging from zero through ten and answers to the second question could be zero or one. –  whuber Jul 27 at 19:16
1  
@whuber: The headline implies it is more likely to have clusters of accidents than widely spaced accidents. Lets operationalize that. Let's say that a cluster is 3 airplane accidents, and a short period of time is 3 months and a long period of time is 3 years. It seems illogical to think that there is a higher probability that 3 accidents will occur within a period of 3 months than within a period of 3 years. Even if we take the first accident as a given, it is illogical to think that 2 more accidents will occur within the next 3 months as compared to within the next 3 years. –  Joel W. Jul 27 at 19:50

The other answers have already dealt with how independent events cluster. (Reading Gleick's Chaos, all those years ago, opened my eyes to this idea.)

But, in fact there is strong evidence that plane crashes are not independent events. Cialdini's Influence has a very good chapter on this (also mentioned here which has a couple of links to data; and I found an excerpt of that part of the book). Obviously this is highly controversial: it is basically saying that the more publicized an air crash is, the more likely it is to influence a pilot (consciously or unconsciously) to crash his plane. But the psychological explanations underlying the hypothesis seem plausible, and the data seems to support it too.

(Links to statistics-based debunking research would be welcome, in the comments.)

share|improve this answer
    
Doesn't say that: says "immediately following certain kinds of highly publicized suicide stories, the number of people who die in commercial-airline crashes increases". –  Scortchi Aug 1 at 11:56
    
The reference for the claim is, I think, Phillips,(1978) "Airplane accident fatalities increase just after newspaper stories about murder and suicide", Science, 201, pp 748-750. The abstract refers to "private, business, and corporate-executive airplanes". –  Scortchi Aug 1 at 12:09
    
Or perhaps this one: Phillips (1980), "Airplane accidents, murder, and the mass media: towards a theory of imitation and suggestion", Social Forces, 58, 4, where "airlines" are mentioned in the abstract. –  Scortchi Aug 1 at 12:13
1  
Altheide (1981), Social Forces, 60, 2 suggests that a "certain kind of highly publicized suicide story" may not have been defined entirely independently of subsequent 'plane crashes - sounds rather like the definition of "famous rabbi". –  Scortchi Aug 1 at 12:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.