Is it still necessary to correct for multiple comparisons/testing if the sample sizes are large?

Question

I was having a discussion with a colleague today regarding corrections for multiple testing. We're planning on running a large number of tests (probably in the hundreds as a rough estimate) and thus I brought up multiple testing and how we should go about it. Specifically the fact that various outcomes are related (not independent) and thus this doesn't meet the assumptions of a Bonferroni, Benjamini-Hochberg or whatever method of adjustment. Their suggestion was that we don't have to worry about it, because the sample size of each test is going to be big enough (we're looking at minimum of n=100 but frequently n=1000+). This is the first time I've ever heard of such an approach and thus was a bit surprised by this. From my research and trying to think it through myself, perhaps their logic was that as n gets larger, sampling error becomes less of an issue, therefore it counteracts any increase in false positives?

My question ultimately is, is my colleague correct in suggesting the large sample sizes are sufficient for handling multiple testing and if so, is my reasoning of it being related to the sampling error accurate?

Answer 1

Think buying hundreds of fair dice. You do not know they are, though, and hence test if each has an expected value of 3.5 points, via throwing each many times (1000+). One of them must come up as "best", and if you do not account for multiple testing, almost certainly statistically significantly so.

Recall that the probability that a true null is rejected should (in practice, that may not be exactly true due to things like asymptotic approximations and finite-sample size distortions) not depend on sample size!

You might then conclude, wrongly (or at least not rightly, in that it is no better, but also no worse than the others), that this is the one you should bring to your next board game.

As to practical significance, this will indeed provide a clue, in that the "winning" one will likely have won with an average of points barely better than 3.5 when you tossed often.

Here is an illustration:

set.seed(1)
dice <- 100
throws <- 1000

tests <- apply(replicate(dice, sample(1:6, throws, 
               replace=T)), 2, 
               function(x) t.test(x, alternative="greater", 
 mu=3.5)) 
# right-tailed test, to look for "better" dice (assuming a 
# game where many points are good, nothing hinges on this)

plot(1:dice, sort(unlist(lapply(1:dice, function(i) 
   tests[[i]]$p.value))))
abline(h=0.05, col="blue")     # significance threshold not 
                       # accounting for multiple testing
abline(h=0.05/dice, col="red") # Bonferroni threshold

max(unlist(lapply(1:dice, function(i) tests[[i]]$estimate))) 
# the sample average of the "winner"

So we see a few "significantly" outperforming dice at level 0.05, but none, in this simulation run, after Bonferroni correction. The "winning" one (last line of the code) however has an average of 3.63, which is, in practice, not too far away from the true expectation 3.5.

We can also run a little Monte Carlo exercise - i.e., the above exercise many times so as to average out any "uncommon" samples that might arise from set.seed(1). We can then also illustrate the effect of varying the number of throws.

# Monte Carlo, with several runs of the experiment:

reps <- 500

mc.func.throws <- function(throws){
  tests <- apply(replicate(dice, sample(1:6, throws, 
                 replace=T)), 2, 
                 function(x) t.test(x, alternative="greater", 
                      mu=3.5)) 
  winning.average <- max(unlist(lapply(1:dice, function(i) 
  tests[[i]]$estimate))) # the sample average of the "winner"
  significant.pvalues <- mean(unlist(lapply(1:dice, 
      function(i) tests[[i]]$p.value)) < 0.05)
  return(list(winning.average, significant.pvalues))
}

diff.throws <- function(throws){
  mc.study <- replicate(reps, mc.func.throws(throws))
  
  average.winning.average <- mean(unlist(mc.study[1,]))
  mean.significant.results <- mean(unlist(mc.study[2,]))
  return(list(average.winning.average, 
 mean.significant.results))
}

throws <- c(10, 50, 100, 500, 1000, 10000)

lapply(throws, diff.throws)

Result:

> unlist(lapply(mc.throws, `[[`, 1))
[1] 4.809200 4.108400 3.927120 3.692292 3.635224 3.542961

> unlist(lapply(mc.throws, `[[`, 2))
[1] 0.04992 0.05134 0.05012 0.04964 0.05006 0.05040

Hence, as predicted, the proportion of statistically significant results is independent of the number of throws (all proportions of $p$-values less than 0.05 are close to 0.05), while the practical significance - i.e., the distance of the average number of points of the "best" one to 3.5 - decreases in the number of throws.

Answer 2

A easy way to make sense of this is in terms of effect sizes, Type 1 errors, Type 2 errors, and Power.

Let's say that you're looking at correlations, and you have $N$ data points.

• Your effect size is the correlation coefficient, $r$.
• Your Type 1 error rate, $\alpha$, is the probability of concluding that there is an effect, $r \neq 0$, when there really is no effect, $r = 0$. We usually use $\alpha = .05$, which means we conclude there is an effect (reject the null hypothesis) when the p-value is $p< .05$. When you conduct multiple tests, you usually adjust for multiple comparisons to keep the Type 1 error rate at $0.05$.
• Your Type 2 error rate, $\beta$, is the probabiliy of failing to conclude that there is an effect when there really is one. Your Power is just the inverse of this, $1 - \beta$.

Now, what happens with different sample sizes?

Here's a table of critical values, from real-statistic.com:

Focus on the third column, where $\alpha = 0.05$. The values in each row show, for each sample size (these rows actually show degrees of freedom, which is just $N - 2$), how strong the correlation $r$ needs to be in order to be significant, $p < .05$:

• When $N = 12$ (df $= 10$), $r \geq 0.57$ is significant.
• When $N = 102$ (df $= 100$), $r \geq 0.19$ is significant.
• When $N = 1002$ (df $= 1000$), $r \geq 0.06$ is significant.

Crucially, the Type 1 error rate is the same in each case, $\alpha = 0.05$. Increasing the sample size has instead made it possible to detect smaller effects, which reduces the Type 2 error rate/boosts power (you're less likely to miss a small but non-zero effect).

Finally, adjusting for multiple comparisons just means using a more conservative threshold for how big $r$ needs to be before you decide there's an effect. In other words, to reduce your Type 1 error rate, you have to increase your Type 2 error rate. Luckily, collecting lots of data means your Type 2 error rate is low, so you can better afford to do this.

So, putting this together:

• More data reduces Type 2 errors, but leaves Type 1 errors unchanged at $\alpha = 0.05$.
• Doing multiple comparisons increases Type 1 errors
• To counteract this, you adjust for multiple comparisons by making your decisions more conservative. This brings the Type 1 error rate back down, at the expense of increasing the Type 2 error rate.

Answer 3

Their suggestion was that we don't have to worry about [multiple testing], because the sample size of each test is going to be big enough (we're looking at minimum of n=100 but frequently n=1000+).

Here's a scenario where your colleagues would be right in practice even if wrong in theory.

Your many tests fall neatly into two categories: $H_0$ is correct (or very close to be) or $H_0$ is extremely inapprioriate. Also, you are not too interested in which tests reject $H_0$ but rather in the overall proportion of accept/reject or you are not too worried about the occasional false positive. In such a case adjusting or not the p-values would make little difference (I still don't see why shouldn't do it though...).

For example, you have a bag of coins, some are legal and some with both sides the same. If you flip each coin a hundred times or so, you are nearly certain to spot the faulty coins by looking at the pvalue, whether you adjust it or not. If the "cost" of a false negative is much higher than a false positive you may just work with nominal p-values.

(I'm not saying your colleagues are right of course, I'm just imaging a scenario that would give them some grounds).