2
$\begingroup$

Regarding meta-analysis, I am wondering whether there is a potential bias due to the ceiling effects (or floor effects) when weighting effect sizes by their sampling error variances.

In case of using standardized mean differences or raw mean scores as effect sizes, we use sampling error variances (by calculating from reported SDs and participant numbers) to weight each effect size. However, SDs tend to become quite small when ceiling effects are observed. I started wondering if just weighting effect sizes by its sampling variances blindly can potentially lead to some kinds of biases for meta-analysis.

Should we care about this? Or, this usually does not bother meta-analysis?

I would appreciate it if you would introduce some articles discussing this issue to me if there is any. Also, I would like to hear ideas on how to deal with this. Just weighting by the number of participants? Or, is there a fancier way to deal with this?

Thank you.

$\endgroup$

1 Answer 1

1
$\begingroup$

One further thing to take into account is that if the primary studies have measures which suffer from floor (or ceiling) effects then the usual assumption that the effect sizes are normally distributed is even less likely to be appropriate than it normally is. If the raw data-sets are available then the problem could be addressed by some form of transformation.

Set against this the likely smaller variances do not seem so important. It is a fact that the variance is smaller and so those studies will receive more weight. If the analyst is seriously worried about any of the estimates including one or more of the sampling variances I would suggest imputing plausible values for the suspicious ones and running sensitivity analyses.

$\endgroup$
1
  • $\begingroup$ Thank you very much for your comments and practical suggestion, @mdewey. I will try the approach you suggested to check the results vary significantly after changing the variance. As for the question, I would like to keep it open for public to see if others offer more perspectives. $\endgroup$ Aug 30, 2018 at 5:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.