Why are correlations acceptable in the small but considered "data dredging" in the large? [duplicate]

Question

This question is going to be a bit more vague than I usually ask on StackExchange sites, but it keeps coming up in my life so I'm going to ask it.

Suppose I have a dataset of N datapoints that are each a vector of K values. Then for each pair of columns of the data I compute some association measure like correlation, and for every pair whose association measure exceeds some threshold I declare, "These fields are associated---something's going on here!"

Then somebody will ask "How big is K?" and I'll say, "K=100,000" and they'll dismiss the correlations as merely a result of "data dredging".

But if I'm making far fewer comparisons (say K=10) then people will be less likely to level that charge against my findings.

So my question is this: why are correlations acceptable in the small but not in the large? Why are the comparisons considered less valid when there are more of them? Shouldn't our inability to trust large numbers of comparisons mean we also shouldn't trust small numbers of comparisons?

Answer 1

Why are the comparisons considered less valid when there are more of them? 

Because the more comparisons you make, the more likely you are to find results just by chance.

For example, if you test the correlation of n uncorrelated variables, the probability that you will find at least one "significant" correlation (at the 5% level) is $1 - (1-0.05)^{\frac{n!}{(n-2)!2!}}$.

In other words, with 100.000 variables, the probability of finding at least one significant correlation is practically 100%. With 10 variables that probability is 90%.

So, from the frequentist point of view, if you test for a lot of correlations, the likelihood of finding at least one strong signal when it does not exist raises and this lowers (according to their view) the strength of your claims.