Examples of costly consequences from improper use of statistical tools

Question

I suspect that most users of statistical tools are ancillary users (folks who have had little to no formal training in statistics). It’s very tempting for researchers and other professionals to apply statistical methods to their data simply because they have seen it “done before” in peer-reviewed papers, grey literature, the web or at a conference. However, doing so without a clear understanding of the required assumptions and the statistical tool’s limitations can lead to erroneous results—errors often unacknowledged!

I find that undergraduate students (particularly in the social and natural sciences) are either unaware of the statistical pitfalls or find these pitfalls inconsequential (the latter being most often the case). Though examples of improper use of statistical tools can be found in many introductory level text books, the web or StackExchange, I have a difficult time finding real-world examples that have had detrimental results (e.g. cost in $, lives impacted and careers lost). To that end, I am looking for real-world examples that highlight the misuse of statistical methods for which:

1. the statistical methods used are typically covered in introductory stats courses (i.e. inferential stats, regressions, etc…)
2. the end result have had costly consequences (dollars lost, lives impacted, careers shattered etc…)
3. the data are readily available for use as working examples in a course (the purpose is to have students work through real world examples that have had real world consequences.)

One non-statistical example I like to bring up to students when discussing the importance of properly defining the units in a research project is the “metric mishap” that led to the loss of a $125M satellite! This usually invokes an :-o factor from the students and seems to have a lasting impression (at least throughout their short academic lives).

Answer 1

A wonderful historical example is afforded by the 1933 publication of Horace Secrist's Triumph of Mediocrity in Business. At the time, Secrist was a well-established statistician, author of a textbook (c. 1919, I recall), well-connected in the American Statistical Association, and head of a statistical research group at Northwestern University. He and his staff had spent the previous decade compiling time series of business data, which are reproduced and painstakingly analyzed in the book. It was meant to be a chef d'oeuvre by an ambitious statistician.

Harold Hotelling's review of the book, which appeared in JASA later that year, pointed out that Secrist had merely documented hundreds of examples of regression to the mean (a fundamental topic in any introductory statistics course today, point #1 of the question). Secrist objected in a published reply. Hotelling's response to that is a classic:

To "prove" such a mathematical result by a costly and prolonged numerical study ... is analogous to proving the multiplication table by arranging elephants in rows and columns, and then doing the same for numerous other kinds of animals. The performance, though perhaps entertaining, and having a certain pedagogical value, is not an important contribution either to zoology or to mathematics.

[JASA v. 29 #186, June 1934, p. 199.]

Secrist seems to have faded quickly from the statistical scene shortly after that ("careers ruined," point #2 in the question). His book is still available. (A few years ago I obtained a nice clean copy, obviously little read, through Interlibrary Loan.) From it you can extract any number of example datasets (point #3 of the question).

Steven Stigler recounts this story in a book and a paper, The history of statistics in 1933.

Answer 2

I'm not sure about data availability, but a great (if that's the right word) example of poor statistics is the Harvard Nurses' Study on the effectiveness of hormone replacement therapy (HRT) in menopausal women.

What's the general idea? The Nurses' Study suggested that HRT was beneficial for post-menopausal women. Turns out that this result arose because the control group was very different from the treatment group and these differences were not account for in the analysis. In subsequent randomized trials, HRT has been linked to cancer, heart attack, stroke, and blood clots. With appropriate corrections, the Nurses' study reveals these patterns as well.

I can't find estimates for US deaths related to HRT, but the magnitude was tens of thousands. One article links 1000 deaths in the UK to HRT.

This New York Times Magazine article provides good statistical background of the issues of confounding present in the study.

There's an academic discussion in this issue of the American Journal of Epidemiology. The articles compare the results of the observational Nurses' study to that of the Women's Health Initiative, based upon randomized trials.

There is also discussion (by many of the same individuals) in an issue of Biometrics See Freedman and Petitti's comment in particular [prepub version].

Answer 3

Seems to me that Wired's take on the 2008 stock market crash might be an informative example. Can't comment on whether it's conclusions are correct or not, but the idea of using correlations over data which are not a representative sample seems like something that might be appropriate to the circumstances you suggest. It's also current, and so might keep them interested.

Answer 4

I thought you might find this Ted Talk interesting and relevant :

• http://www.ted.com/talks/ben_goldacre_battling_bad_science.html