This question is going to be a bit more vague than I usually ask on StackExchange sites, but it keeps coming up in my life so I'm going to ask it.
Suppose I have a dataset of N datapoints that are each a vector of K values. Then for each pair of columns of the data I compute some association measure like correlation, and for every pair whose association measure exceeds some threshold I declare, "These fields are associated---something's going on here!"
Then somebody will ask "How big is K?" and I'll say, "K=100,000" and they'll dismiss the correlations as merely a result of "data dredging".
But if I'm making far fewer comparisons (say K=10) then people will be less likely to level that charge against my findings.
So my question is this: why are correlations acceptable in the small but not in the large? Why are the comparisons considered less valid when there are more of them? Shouldn't our inability to trust large numbers of comparisons mean we also shouldn't trust small numbers of comparisons?