# Examples of costly consequences from improper use of statistical tools

I suspect that most users of statistical tools are ancillary users (folks who have had little to no formal training in statistics). It’s very tempting for researchers and other professionals to apply statistical methods to their data simply because they have seen it “done before” in peer-reviewed papers, grey literature, the web or at a conference. However, doing so without a clear understanding of the required assumptions and the statistical tool’s limitations can lead to erroneous results—errors often unacknowledged!

I find that undergraduate students (particularly in the social and natural sciences) are either unaware of the statistical pitfalls or find these pitfalls inconsequential (the latter being most often the case). Though examples of improper use of statistical tools can be found in many introductory level text books, the web or StackExchange, I have a difficult time finding real-world examples that have had detrimental results (e.g. cost in $, lives impacted and careers lost). To that end, I am looking for real-world examples that highlight the misuse of statistical methods for which: 1. the statistical methods used are typically covered in introductory stats courses (i.e. inferential stats, regressions, etc…) 2. the end result have had costly consequences (dollars lost, lives impacted, careers shattered etc…) 3. the data are readily available for use as working examples in a course (the purpose is to have students work through real world examples that have had real world consequences.) One non-statistical example I like to bring up to students when discussing the importance of properly defining the units in a research project is the “metric mishap” that led to the loss of a$125M satellite! This usually invokes an :-o factor from the students and seems to have a lasting impression (at least throughout their short academic lives).

-
Another non-statistical example from Edward Tufte, Powerpoint does Rocket Science. Although it is bit more closely related to logical progression from statistical thinking in general than the metric mishap you mention. Also are you familiar with this book, The Cult of Statistical Significance? – Andy W Oct 19 '11 at 14:58
@AndyW, I'm not familiar with "The Cult of Statistical Significance". Do you know if elements 2/ and 3/ in my question are addressed in that book? – MannyG Oct 19 '11 at 15:04
I don't know about 3, but if you read the book review I linked to it would answer your question 2 (or read the remaining title of the book!) Indeed the whole book is intended to be about your question #2 in reference to interpreting significance tests. – Andy W Oct 19 '11 at 15:10
@AndyW that is the very book I was going to mention. – Peter Flom Oct 19 '11 at 15:13
@AndyW, though the review you link references one of the book's real-world example of improper use of stats, it's not clear to me if the consequential cost is addressed. If consequential costs are addressed in the book are they based on independent analyses or the authors'subjective opinion? – MannyG Oct 19 '11 at 15:31

A wonderful historical example is afforded by the 1933 publication of Horace Secrist's Triumph of Mediocrity in Business. At the time, Secrist was a well-established statistician, author of a textbook (c. 1919, I recall), well-connected in the American Statistical Association, and head of a statistical research group at Northwestern University. He and his staff had spent the previous decade compiling time series of business data, which are reproduced and painstakingly analyzed in the book. It was meant to be a chef d'oeuvre by an ambitious statistician.

Harold Hotelling's review of the book, which appeared in JASA later that year, pointed out that Secrist had merely documented hundreds of examples of regression to the mean (a fundamental topic in any introductory statistics course today, point #1 of the question). Secrist objected in a published reply. Hotelling's response to that is a classic:

To "prove" such a mathematical result by a costly and prolonged numerical study ... is analogous to proving the multiplication table by arranging elephants in rows and columns, and then doing the same for numerous other kinds of animals. The performance, though perhaps entertaining, and having a certain pedagogical value, is not an important contribution either to zoology or to mathematics.

[JASA v. 29 #186, June 1934, p. 199.]

Secrist seems to have faded quickly from the statistical scene shortly after that ("careers ruined," point #2 in the question). His book is still available. (A few years ago I obtained a nice clean copy, obviously little read, through Interlibrary Loan.) From it you can extract any number of example datasets (point #3 of the question).

Steven Stigler recounts this story in a book and a paper, The history of statistics in 1933.

-
 +1 for satisfying #1, #2 and #3 and the nice narrative. – MannyG Oct 21 '11 at 23:45

I'm not sure about data availability, but a great (if that's the right word) example of poor statistics is the Harvard Nurses' Study on the effectiveness of hormone replacement therapy (HRT) in menopausal women.

What's the general idea? The Nurses' Study suggested that HRT was beneficial for post-menopausal women. Turns out that this result arose because the control group was very different from the treatment group and these differences were not account for in the analysis. In subsequent randomized trials, HRT has been linked to cancer, heart attack, stroke, and blood clots. With appropriate corrections, the Nurses' study reveals these patterns as well.

I can't find estimates for US deaths related to HRT, but the magnitude was tens of thousands. One article links 1000 deaths in the UK to HRT.

This New York Times Magazine article provides good statistical background of the issues of confounding present in the study.

There's an academic discussion in this issue of the American Journal of Epidemiology. The articles compare the results of the observational Nurses' study to that of the Women's Health Initiative, based upon randomized trials.

There is also discussion (by many of the same individuals) in an issue of Biometrics See Freedman and Petitti's comment in particular [prepub version].

-
I'd argue against the use of this example actually. There's been more work since 2005, especially by Miguel Hernan, see Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease, Epidemiology (2008). The conclusion: "In summary, our findings suggest that the discrepancies between the WHI and NHS ITT estimates could be largely explained by differences in the distribution of time since menopause and length of follow-up. Residual confounding for the effect of therapy initiation in the NHS seems to play little role." – EpiGrad Oct 19 '11 at 19:41
Regardless of how one feels about said studies, the differences between them are trickier and more complicated than what's probably useful in the OP's proposed setting. – EpiGrad Oct 19 '11 at 19:42
@EpiGrad, This certainly isn't my field and I'm sure that you know more about this particular example than I do. But, I think that the paper that you cite makes the point that those other papers do. In the OSALRE paper, they throw out women from the NHS study who don't meet the criteria of the WHI study. The proportion of women discarded must vary across the NHS treatment and control groups (or else the results wouldn't change). They are thus undoing the selection problem found in the NHS study. [[Continued]] – Charlie Oct 19 '11 at 20:18
The paper I liked suggests, at least to me, something more along the lines of "Make sure you're asking the same question" rather than a problem of outright confounding. Don't get me wrong, the NHS/WHI issue is extremely interesting as a statistical and public health practice question. It's just more complex than the initial disagreement seems to suggest, and I think that makes it somewhat inappropriate for Point 1 of the OP's request. Point 3 is also right out. – EpiGrad Oct 19 '11 at 20:39
@EpiGrad, Fair enough. But I seriously doubt that you'll find a study that has a glaring statistical error that doesn't require some digging to understand that had widespread, substantial consequences. Maybe other respondents will kill my optimism in researchers, though (heh). – Charlie Oct 19 '11 at 21:17