Assume you conduct 100 significance tests in a single study (maybe something like genome-wide association study). In order to address the issue of multiple testing and to keep the family-wise error rate at 5%, the p-values are adjusted, e.g., using Bonferroni or FDR (Benjamini-Hochberg). Are significance tests that are still significant after adjustment, over-estimating the true effect? A quick simulation in R seems to suggest "yes" - at least under some conditions. In my simulation, small effects showed this exaggeration, but large effects do not. The reason behind this is that the large effec always showed a significant result for the sample size I have chosen, so conditioning on being significant did not change anything.

Is there a more formal explanation of this, and maybe some more formal literature on this issue? I am aware of Gelman's Type M error (error of magnitude), but am curious what else is out there.

enter image description here R code below:

effest <- function() {
 n <- 1000

 #random treatment
 tr <- rbinom(n,1,.5)
#two true nulls
x1 <- rnorm(n,0,1)
x2 <- rnorm(n,0,1)

#two false nulls, variances of errors are adjusted so that all x's have total variance of 1
#x3 has effect of .2
#x4 has effect of .5
x3 <- .2*tr + rnorm(n,0,.98)
x4 <- .5*tr + rnorm(n,0,.865)

t1 <- t.test(x1~tr)
t2 <- t.test(x2~tr)
t3 <- t.test(x3~tr)
t4 <- t.test(x4~tr)

ps <- c(t1$p.value,t2$p.value,t3$p.value,t4$p.value,

res <- data.frame(raply(5000,effest(),.progress = "text"))
names(res) <- c("p1","p2","p3","p4","m10","m11","m20","m21","m30","m31","m40","m41")

res$md1 <- res$m11-res$m10
res$md2 <- res$m21-res$m20
res$md3 <- res$m31-res$m30
res$md4 <- res$m41-res$m40

#pvalues of true null

#pvalues of false nulls

#treatment effects of true nulls, across all reps

#treatment effects of false nulls, across all reps

#treatment effects of true nulls, across sig reps
hist(res$md1[res$p1 <.05])
abline(v=mean(res$md1[res$p1 <.05]),col="red")
hist(res$md2[res$p2 <.05])
abline(v=mean(res$md2[res$p2 <.05]),col="red")

#treatment effects of false nulls, across sig reps
hist(res$md3[res$p3 <.05])
abline(v=mean(res$md3[res$p3 <.05]),col="red")
hist(res$md4[res$p4 <.05])
abline(v=mean(res$md4[res$p4 <.05]),col="red")

#treatment effects of true nulls, across sig reps using stringent p < .001
hist(res$md1[res$p1 <.001])
abline(v=mean(res$md1[res$p1 <.001]),col="red")
hist(res$md2[res$p2 <.001])
abline(v=mean(res$md2[res$p2 <.001]),col="red")

#treatment effects of false nulls, across sig reps using stringent p < .001
hist(res$md3[res$p3 <.001])
abline(v=mean(res$md3[res$p3 <.001]),col="red")
hist(res$md4[res$p4 <.001])
abline(v=mean(res$md4[res$p4 <.001]),col="red")
  • $\begingroup$ Isn't it true that all the effect sizes are the same, and the significant ones comprise the top tier of those effects. Thus if you apply a multiplicity adjustment, you are raising the threshold for which effects can be declared significant. You get a smaller number of significant effects, therefore the average effect size of those will be larger because you're discarding some of the smaller effects from the group of unadjusted significant ones. $\endgroup$
    – Russ Lenth
    Commented Feb 18, 2017 at 2:06
  • $\begingroup$ Yes, the effect sizes are estimated without bias, as would be expected. Due to sampling some are a bit larger, some are a bit smaller, but on average they are correct. Conditioning on being significant, the average of these effect sizes is inflated. Conditioning on being significant using an even more stringent criterion, inflates them even more. Thus, it seems that if one is using a stringent adjustment to declare something a significant effect, this detected effect will very likely be too large. $\endgroup$ Commented Feb 18, 2017 at 2:12

1 Answer 1


I wonder if this will answer your question...

Suppose you have 100 effects of absolute effect sizes $e_1, e_2, \ldots, e_{100}$, and let their order statistics be $e_{(1)} \le e_{(2)} \le \cdots \le e_{(100)}$.

Suppose that 20 of them are significant based on an unadjusted test, so the average effect size of those is $\bar e_S^U$, the average of $e_{(81)}$ through $e_{(100)}$. Similarly, the nonsignificant ones have average effect size of $\bar e_N^U$, the average of $e_{(1)}$ through $e_{(80)}$.

Now, consider a multiplicity-adjusted test; its critical value is higher so that fewer effects will be found significant. Suppose there are in fact 10 of them. Then the average effect size of the significant ones based on then adjusted test is $\bar e_S^A$, the average of $e_{(91)}$ through $e_{(100)}$; and the average effect size of the nonsignificant ones is $\bar e_N^A$, the average of $e_{(1)}$ through $e_{(90)}$.

Note that

  • $\bar e_S^A \ge \bar e_S^U$, because $\bar e_S^A$ excludes the 10 smallest that are included in $\bar e_S^U$. But also,
  • $\bar e_N^A \ge \bar e_N^U$, because $\bar e_N^A$ includes 10 effects that are larger than any in $\bar e_N^U$.

So both averages go up when you use a multiplicity adjustment, even though the average of them all, $\bar e$, is unchanged. This seems counter-intuitive, but there you have it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.