Potential bias of effect sizes due to ceiling (or floor) effects when weighting effect sizes by their variance for meta-analysis

Question

Regarding meta-analysis, I am wondering whether there is a potential bias due to the ceiling effects (or floor effects) when weighting effect sizes by their sampling error variances.

In case of using standardized mean differences or raw mean scores as effect sizes, we use sampling error variances (by calculating from reported SDs and participant numbers) to weight each effect size. However, SDs tend to become quite small when ceiling effects are observed. I started wondering if just weighting effect sizes by its sampling variances blindly can potentially lead to some kinds of biases for meta-analysis.

Should we care about this? Or, this usually does not bother meta-analysis?

I would appreciate it if you would introduce some articles discussing this issue to me if there is any. Also, I would like to hear ideas on how to deal with this. Just weighting by the number of participants? Or, is there a fancier way to deal with this?

Thank you.

Answer 1

One further thing to take into account is that if the primary studies have measures which suffer from floor (or ceiling) effects then the usual assumption that the effect sizes are normally distributed is even less likely to be appropriate than it normally is. If the raw data-sets are available then the problem could be addressed by some form of transformation.

Set against this the likely smaller variances do not seem so important. It is a fact that the variance is smaller and so those studies will receive more weight. If the analyst is seriously worried about any of the estimates including one or more of the sampling variances I would suggest imputing plausible values for the suspicious ones and running sensitivity analyses.