War stories where wrong decisions were made based on statistical information?

Question

I think it is fair to say statistics is an applied science so when averages and standard deviations are calculated it is because someone is looking to make some decisions based on those numbers.

Part of being a good statistician then I would hope is being able to "sense" when the sample data can be trusted and when some statistical test is completely misrepresenting the true data we're interested in. Being a programmer that is interested in analysis of big data sets I'm relearning some statistics and probability theory but I can't shake this nagging feeling that all the books I've looked at are kind of like politicians that get up on stage and say a whole bunch of things and then append the following disclaimer at the end of their speech:

Now, I'm not saying that this is good or bad but the numbers say it's good so you should vote for me anyway.

Maybe you get that but maybe you don't so here's a question. Where do I go to find war stories by statisticians where some decisions were based on some statistical information that later turned out to be completely wrong?

Answer 1

This isn't exactly what you're asking for, but I think you'd like David Freedman's work. He calls BS on misapplication of statistical tests, etc... See here: http://www.stat.berkeley.edu/~freedman/. One of my favorites is “What is the probability of an earthquake?”.

Answer 2

You might check out a recent presentation on SSRN by Bernard Black, "Bloopers: How (Mostly) Smart People Get Causal Inference Wrong." http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1663404

I will say that I also admire David Freedman and appreciate his work. Though I was a UC Berkeley grad student while he was here, he passed away before I had a chance to take his course. You might have a look at his collected works edited by a few other Berkeley professors: "Statistical Models and Causal Inference: A Dialogue with the Social Sciences."

Answer 3

This thread is now ancient but it may still be worth posting the results of a recent study titled, A bot crawled thousands of studies looking for simple math errors. The results are concerning. It's not exactly a war story but it does illustrate the rampant errors inherent in published, peer reviewed papers. http://www.vox.com/science-and-health/2016/9/30/13077658/statcheck-psychology-replication

Answer 4

The sad case of Sally Clark springs to mind.

In 1999, she was wrongfully convicted of murdering her two sons after it was erroneously concluded by Professor Sir Roy Meadow that the chances of both of her sons dying from sudden infant death syndrome (SIDS) were 1 in 73 million, and his now discredited eponymous law:

One is a tragedy, two is suspicious and three is murder unless there is proof to the contrary.

The Royal Statistical Society criticised the statistical evidence on two counts:

1. The incorrect assumption of independence between SIDS in siblings. This made the calculation used by Meadow - squaring the probability for a single SIDS incident - invalid.
2. A misrepresentation of the statistics giving rise to the prosecutor's fallacy.

Furthermore, there were concerns raised by Ray Hill about the quality of the underlying data used to compute the chance of a single SIDS event.

After two appeals, Clark was eventually acquitted, but the experience of losing both sons, and the miscarriage of justice left her psychologically scarred and she died of alcohol poisoning in 2007.