# Effect size - is correction for multiple comparison needed?

I have a general question on whether effect sizes need to be corrected for multiple comparisons. I need to conduct 8 t-tests, in order to test whether values from 8 different conditions different significant from 0. I know that are procedures that allow an adjustment of the p-value according to the number of tests conducted (e.g., Bonferroni). I was wondering whether a similar approach needs to be employed for effect sizes, and if so which would be the best option.

The 8 conditions I am testing reflect the cells of my 2x2x2 within-subject design. The issue with the multiple comparison appears twice: 1. When conducting an ANOVA, I find a 3-way interaction and need to conduct post-hoc comparisons. I know how to adjust the p-values, but not the effect size estimate (indeed, I asked whether this is in principle possible at all); 2. When I want to test whether each of 8 conditions (2x2s2) is significantly different from zero. For both options I should then just adjust the CI, but not the estimates?

• Based on the response and discussion to date, can you specify if you are thinking about the confidence intervals around the estimated effect size statistic, or if you are thinking about the estimated effect size statistic ? Can we assume that you are using a form of Cohen's d that is appropriate in the context of a one-sample t test ? Feb 18, 2022 at 17:00
• The 8 conditions I am testing reflect the cells of my 2x2x2 within-subject design. The issue with the multiple comparison appears twice: 1. When conducting an ANOVA, I find a 3-way interaction and need to conduct post-hoc comparisons. I know how to adjust the p-values, but not the effect size estimate (indeed, I asked whether this is in principle possible at all); 2. When I want to test whether each of 8 conditions (2x2s2) is significantly different from zero. For both options I should then just adjust the CI, but not the estimates? Feb 21, 2022 at 9:07

What people typically refer to as 'the effect size' for some data is a point estimate. Just as you can get a point estimate of a regression slope and a confidence interval around that, you can get an interval estimate for an effect size. For example, from the data for a t-test, you can compute a standardized mean difference, and you can get a 95% CI around it.

If you have multiple comparisons, there are several ways you can think about the individual comparisons within the set. You can treat them individually, or you might be concerned about the probability of Type I errors within the set as a whole (or some gradation in between). If the latter were your situation, you might use something like a Bonferroni correction. This could be done to alpha, your threshold for significance, or you could do it to the p-value itself. You can also use the adjusted thresholds to compute a Bonferroni-adjusted 95% CI.

If you are computing interval estimates for multiple effect sizes, and you are worried about the potential for multiplicity to influence the larger picture, you can make exactly the same kind of Bonferroni-adjusted 95% CI for the effect sizes. For instance, if you had 2 effect sizes, a 97.5% (1-.05/2) CI would be the Bonferroni-adjusted 95% CI.

• I took it that the question is about adjusting point estimates, not the confidence intervals around these estimates. Feb 18, 2022 at 15:45
• @num_39, that may well have been what the OP was wondering about, but you don't adjust the point estimates. You might make adjustments, if you want, but you would make them to the CIs. Feb 18, 2022 at 15:49
• yes, but then that's the answer. In any case, between your answer and these comments, the OP should have an answer. :) Feb 18, 2022 at 15:52
• Though I can imagine some sort of Bayesian adjustments toward zero such that when you adjust the confidence intervals, the estimated effect size is further adjusted toward zero. Feb 18, 2022 at 15:55
• Although I also think of "effect sizes" as point estimates, the Wikipedia entry shows that many think of "effect sizes" instead as point estimates "standardized" by a standard deviation or some such, like Cohen's d. I suspect that the latter is the type of "effect size" the OP had in mind. This answer also shows how to do such adjustments. +1.
– EdM
Feb 18, 2022 at 16:21

It sounds like you are in a play-the-winner scenario: You pick amongst several effect sizes and only select the largest ones (or only those that are "statistically significant) or something like that. In that case, your estimate of the effect size for the picked comparison will indeed be biased upwards.

What can one do about it? As others pointed out, you can valid confidence intervals by adjusting the confidence level in line with Bonferroni (i.e. using two-sided 99.375% confidence intervals instead of 95% ones), but I'm not sure how helpful that truly is (depends somewhat on what you do with these next). An experimental solution is to run another experiment afterwards to get a second unbiased estimate. Other ideas of addressing this include using Bayesian hierarchical models such as with a regularized horseshoe prior that both shrink point estimates and provide credible intervals (where the main challenge is picking a suitable prior = specifying the "needed" amount of shrinkage).

• Thanks for the suggestion. I was wondering whether CI can be corrected also using False Discovery Rate, that should be possible right? Feb 21, 2022 at 14:25
• Not sure on the options there. There may be methods out there, but getting CIs that line up with test decisions can be tricky. Feb 21, 2022 at 14:32