Effect size - is correction for multiple comparison needed? I have a general question on whether effect sizes need to be corrected for multiple comparisons. I need to conduct 8 t-tests, in order to test whether values from 8 different conditions different significant from 0. I know that are procedures that allow an adjustment of the p-value according to the number of tests conducted (e.g., Bonferroni). I was wondering whether a similar approach needs to be employed for effect sizes, and if so which would be the best option.
The 8 conditions I am testing reflect the cells of my 2x2x2 within-subject design. The issue with the multiple comparison appears twice: 1. When conducting an ANOVA, I find a 3-way interaction and need to conduct post-hoc comparisons. I know how to adjust the p-values, but not the effect size estimate (indeed, I asked whether this is in principle possible at all); 2. When I want to test whether each of 8 conditions (2x2s2) is significantly different from zero. For both options I should then just adjust the CI, but not the estimates?
 A: What people typically refer to as 'the effect size' for some data is a point estimate.  Just as you can get a point estimate of a regression slope and a confidence interval around that, you can get an interval estimate for an effect size.  For example, from the data for a t-test, you can compute a standardized mean difference, and you can get a 95% CI around it.
If you have multiple comparisons, there are several ways you can think about the individual comparisons within the set.  You can treat them individually, or you might be concerned about the probability of Type I errors within the set as a whole (or some gradation in between).  If the latter were your situation, you might use something like a Bonferroni correction.  This could be done to alpha, your threshold for significance, or you could do it to the p-value itself.  You can also use the adjusted thresholds to compute a Bonferroni-adjusted 95% CI.
If you are computing interval estimates for multiple effect sizes, and you are worried about the potential for multiplicity to influence the larger picture, you can make exactly the same kind of Bonferroni-adjusted 95% CI for the effect sizes.  For instance, if you had 2 effect sizes, a 97.5% (1-.05/2) CI would be the Bonferroni-adjusted 95% CI.
A: It sounds like you are in a play-the-winner scenario: You pick amongst several effect sizes and only select the largest ones (or only those that are "statistically significant) or something like that. In that case, your estimate of the effect size for the picked comparison will indeed be biased upwards.
What can one do about it? As others pointed out, you can valid confidence intervals by adjusting the confidence level in line with Bonferroni (i.e. using two-sided 99.375% confidence intervals instead of 95% ones), but I'm not sure how helpful that truly is (depends somewhat on what you do with these next). An experimental solution is to run another experiment afterwards to get a second unbiased estimate. Other ideas of addressing this include using Bayesian hierarchical models such as with a regularized horseshoe prior that both shrink point estimates and provide credible intervals (where the main challenge is picking a suitable prior = specifying the "needed" amount of shrinkage).
