# Under torture, the data may yield false confessions. Examples?

My statistics teacher repeated the statement "Under torture, the data may yield false confessions" several times, but without giving a concrete example of what it means. Can you give some good examples that show the aim of the statement?

• Not the question, but all supposed jokes about torture are tasteless in my view. – Nick Cox Jan 15 '18 at 17:06
• It may mean a bunch of things, e.g. this may be a comment related to overfitting. – tagoma Jan 15 '18 at 17:08
• Have a google.scholar for +"dead salmon" +fmri or for +zodiac +"data dredging". – Björn Jan 15 '18 at 17:29
• – gung Jan 15 '18 at 17:57
• A while ago one hot question asked for famous statistics quotes. I provided this one and got heavily downvoted. – Vladislavs Dovgalecs Jan 15 '18 at 18:15

What is meant by "torture" is ambiguous. However, I believe that subjecting data to procedures for which it is not intended is a form of this torture. Anscombe's quartet is a classic example of this, subjecting four sets of data to linear regression when three of them clearly do not fit the assumptions.

A second kind of data torture is overfitting, defined many places including this one.

• Applying methods to data for which they are not suited seems to me a different problem (and one perhaps even more common). The question seems to be driving at analyses that are over-elaborate and/or too much repeated, which I don't think is exemplified by the Anscombe quartet. – Nick Cox Jan 16 '18 at 17:56
• @NickCox The question seems to be driving at the meaning of the quote. Unless the original author, of the quote, describes the intent, then anything we think is just opinion. It's actually most likely that the author of the quote wanted us to imagine a host of impropriety and apply the quote widely, to learn prudence and caution in data analysis. – Mark Jones Jr. Jan 17 '18 at 0:56
• Naturally I can't give privileged insight into what the author really intended, but the essence of torture seems to include the use of extreme measures. Thus I can't see your example as "torture"; it is a simple illustration of how standard methods can miss the message in the data. – Nick Cox Jan 17 '18 at 1:02

This is known as "data dredging". You can read wikipedia.

The simplest case is when testing 1000 independent hypothesis each with type I risk 1%, you can be almost use one of them will be positive, as a matter of chance.

• Your statement, while true, probably doesn't apply in practice because $1000$ dependent (to varying degrees) hypothesis are usually what is actually done. – probabilityislogic Jan 16 '18 at 0:45

I prefer the term "cherry picking", which may result from testing tons of hypotheses using unadjusted alpha-level (same as @Benoit Sanchez's).