Can someone explain this cartoon? 
I ran 160 regressions on various combinations of data sets, predictors, and dependent variables. I am now trying to sift through the results and separate good models which are "real" versus those which are due to random chance. My professor sent me this cartoon. It seems to be implying that by using $\alpha = 0.05$, you are highly likely to get significance in 20 data sets.
Questions:


*

*Does random data have a 0.05 probability of being significant at $\alpha = 0.05$?

*In other words:
$$\Pr(sig>0.05|n=20) = 1 -(1-0.05)^{20} = 0.6415$$
Not exactly a guarantee as suggested by the cartoon.
Is the above calculation true for regressions?


*Is there a word for this "random correlation" so I can do further research?

 A: Your calculation is right assuming the p-values arise from data that are independent and arise from sufficiently large datasets (so that the p-values are really approximately uniformly distributed under the null hypothesis). If the data are not independent (e.g. when all people who were randomly assigned to eat a certain color of jelly bean are compared to a single group of people assigned to eat no jelly beans, or if people that ate red jelly beans are compared to people that did not eat jelly beans both in terms of occurrence of acne, cancer or death), things are also more complicated.
Thus, you are right that there is no guarantee that amongst 20 comparisons conducted under the null hypothesis there would be at least one type I error and the familywise type I error rate you calculated is exactly right. However, in case of 160 comparisons the familywise type I error rate should be very close to 100%. There are a number of possible ways to deal with this type of multiplicity. These include testing procedures that control the familywise type I error rate (e.g. the Bonferroni-Holm procedure) or the false discovery rate. I have also seen some Bayesians argue for (implicit) shrinkage approaches using some kind of hierarchical Bayesian model and there are almost certainly further things one could do.
