# Can someone explain this cartoon? [duplicate]

I ran 160 regressions on various combinations of data sets, predictors, and dependent variables. I am now trying to sift through the results and separate good models which are "real" versus those which are due to random chance. My professor sent me this cartoon. It seems to be implying that by using $\alpha = 0.05$, you are highly likely to get significance in 20 data sets.

Questions:

1. Does random data have a 0.05 probability of being significant at $\alpha = 0.05$?

2. In other words:

$$\Pr(sig>0.05|n=20) = 1 -(1-0.05)^{20} = 0.6415$$

Not exactly a guarantee as suggested by the cartoon.

Is the above calculation true for regressions?

1. Is there a word for this "random correlation" so I can do further research?
• 1. Yes, or more generally "if the null hypothesis is true." Random data do not always mean zero difference or zero correlation (aka you can have randomly generated data that are correlated at r = 0.8); 2. Why are you computing them as if they are dependent events? In the cartoon all 20 tests are independent events. 3. Usually it's called "Type I error" or "false positive." Jan 25, 2016 at 15:36
• **Your professor is using a humorous cartoon to warn you about the trap you have stepped into. Don't try to argue with the cartoon instead of heading the warning. You're right to want to learn more about the problem: semanticommunity.info/@api/deki/files/30744/… ** Jan 25, 2016 at 16:35
• Thanks @Wayne, that presentation is extremely clarifying. Jan 25, 2016 at 20:29
• Argh, I though it was a paper, but it's a slide deck. Should've looked closer. I hope you did get something out of it, since a lot of detail is left out of the slides to give the speaker room to talk. Bottom line is, even with the binomial math you did -- which isn't quite what you think it is -- the odds are almost 2:1 (0.6415 v 0.3585) that you have a problem. Would you listen to a lecture where someone started with, "the odds are 1 out of 3 that what I found might be real!" (Apply your math to your 160 regressions and the odds explode.) Jan 25, 2016 at 20:52
• @Wayne ,you hinted that the binomial math I did is not right. Why? Is is because the trials are not independent? I can see how a regression of X1 vs Y1 would not be independent of a regression of X1 vs Y2. If this is the issue, is there a way quantify the probability of getting a Type I error in n trials? Jan 26, 2016 at 14:49