Does the use of Type I error adjustment, inflates effect sizes of significant results?

Question

Assume you conduct 100 significance tests in a single study (maybe something like genome-wide association study). In order to address the issue of multiple testing and to keep the family-wise error rate at 5%, the p-values are adjusted, e.g., using Bonferroni or FDR (Benjamini-Hochberg). Are significance tests that are still significant after adjustment, over-estimating the true effect? A quick simulation in R seems to suggest "yes" - at least under some conditions. In my simulation, small effects showed this exaggeration, but large effects do not. The reason behind this is that the large effec always showed a significant result for the sample size I have chosen, so conditioning on being significant did not change anything.

Is there a more formal explanation of this, and maybe some more formal literature on this issue? I am aware of Gelman's Type M error (error of magnitude), but am curious what else is out there.

R code below:

effest <- function() {
 n <- 1000

 #random treatment
 tr <- rbinom(n,1,.5)
#two true nulls
x1 <- rnorm(n,0,1)
x2 <- rnorm(n,0,1)

#two false nulls, variances of errors are adjusted so that all x's have total variance of 1
#x3 has effect of .2
#x4 has effect of .5
x3 <- .2*tr + rnorm(n,0,.98)
x4 <- .5*tr + rnorm(n,0,.865)

t1 <- t.test(x1~tr)
t2 <- t.test(x2~tr)
t3 <- t.test(x3~tr)
t4 <- t.test(x4~tr)

ps <- c(t1$p.value,t2$p.value,t3$p.value,t4$p.value,
      as.numeric(t1$estimate),as.numeric(t2$estimate),as.numeric(t3$estimate),as.numeric(t4$estimate))
return(ps)
}

library(plyr)
res <- data.frame(raply(5000,effest(),.progress = "text"))
names(res) <- c("p1","p2","p3","p4","m10","m11","m20","m21","m30","m31","m40","m41")

res$md1 <- res$m11-res$m10
res$md2 <- res$m21-res$m20
res$md3 <- res$m31-res$m30
res$md4 <- res$m41-res$m40

par(mfrow=c(2,2))
#pvalues of true null
hist(res$p1)
hist(res$p2)

#pvalues of false nulls
hist(res$p3)
hist(res$p4)


#treatment effects of true nulls, across all reps
hist(res$md1)
abline(v=mean(res$md1),col="red")
hist(res$md2)
abline(v=mean(res$md2),col="red")

#treatment effects of false nulls, across all reps
hist(res$md3)
abline(v=mean(res$md3),col="red")
hist(res$md4)
abline(v=mean(res$md4),col="red")


#treatment effects of true nulls, across sig reps
hist(res$md1[res$p1 <.05])
abline(v=mean(res$md1[res$p1 <.05]),col="red")
hist(res$md2[res$p2 <.05])
abline(v=mean(res$md2[res$p2 <.05]),col="red")

#treatment effects of false nulls, across sig reps
hist(res$md3[res$p3 <.05])
abline(v=mean(res$md3[res$p3 <.05]),col="red")
hist(res$md4[res$p4 <.05])
abline(v=mean(res$md4[res$p4 <.05]),col="red")


#treatment effects of true nulls, across sig reps using stringent p < .001
hist(res$md1[res$p1 <.001])
abline(v=mean(res$md1[res$p1 <.001]),col="red")
hist(res$md2[res$p2 <.001])
abline(v=mean(res$md2[res$p2 <.001]),col="red")

#treatment effects of false nulls, across sig reps using stringent p < .001
hist(res$md3[res$p3 <.001])
abline(v=mean(res$md3[res$p3 <.001]),col="red")
hist(res$md4[res$p4 <.001])
abline(v=mean(res$md4[res$p4 <.001]),col="red")

Answer 1

I wonder if this will answer your question...

Suppose you have 100 effects of absolute effect sizes $e_1, e_2, \ldots, e_{100}$, and let their order statistics be $e_{(1)} \le e_{(2)} \le \cdots \le e_{(100)}$.

Suppose that 20 of them are significant based on an unadjusted test, so the average effect size of those is $\bar e_S^U$, the average of $e_{(81)}$ through $e_{(100)}$. Similarly, the nonsignificant ones have average effect size of $\bar e_N^U$, the average of $e_{(1)}$ through $e_{(80)}$.

Now, consider a multiplicity-adjusted test; its critical value is higher so that fewer effects will be found significant. Suppose there are in fact 10 of them. Then the average effect size of the significant ones based on then adjusted test is $\bar e_S^A$, the average of $e_{(91)}$ through $e_{(100)}$; and the average effect size of the nonsignificant ones is $\bar e_N^A$, the average of $e_{(1)}$ through $e_{(90)}$.

Note that

• $\bar e_S^A \ge \bar e_S^U$, because $\bar e_S^A$ excludes the 10 smallest that are included in $\bar e_S^U$. But also,
• $\bar e_N^A \ge \bar e_N^U$, because $\bar e_N^A$ includes 10 effects that are larger than any in $\bar e_N^U$.

So both averages go up when you use a multiplicity adjustment, even though the average of them all, $\bar e$, is unchanged. This seems counter-intuitive, but there you have it.