I have to conduct a meta-analysis. Following problem occured: a single study have measured Quality of Life and therefore, used two questionnaires. Now is my question whether I have two choose one of the instrument to include in meta-analysis or is there a possibility to combine the means of the two scales to provide one standard mean difference in the meta-analysis? THANK YOU!!!!


At least 3 different approaches are possible:

  1. You compute two standardized mean differences for this study and include both of these values in the meta-analysis.

    Since the two effects were computed based on the same sample, they are not independent. Therefore, you have to account for and incorporate the degree of dependence between the two estimates in the analysis. This means that you will have to calculate the covariance between the two estimates, which requires knowing the size of the correlation of the raw scores for the two instruments. This is typically not reported, so at that point, you would have to 'guestimate' a reasonable value for the correlation and then conduct a sensitivity analysis in the end (by varying the guestimated value within a reasonable range and then checking whether the conclusions remain unchanged).

    The equation for the covariance can be found in:

    Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York: Russell Sage Foundation.

    The model required for the analysis is also a bit more complex. If you are using R, you could take a look at: http://www.metafor-project.org/doku.php/analyses:gleser2009#multiple-endpoint_studies where I actually describe this type of analysis based on the Gleser & Olkin (2009) chapter (although in practice, one may consider adding random effects to the model to model heterogeneity).

    While this approach maximizes the amount of information extracted from the studies, it comes with a lot of additional complexities that are hardly worth it if you only have one or two studies like that.

  2. You compute two standardized mean differences for this study and average them.

    This is in fact a common practice in meta-analysis. From a purely statistical perspective, this approach is not quite right. The average itself is fine, but the sampling variance of the average is typically not computed in the right way. Computing this correctly again requires knowing the correlation of the raw scores. So, while this approach avoids the additional complexities of having to use a model that accommodates multiple effects for the same study, if you want to do this in the correct way, you still need information that is not typically reported and extra steps to check for the sensitivity of the conclusions to the 'guestimated' size of the correlation.

  3. You set up a hierarchy of 'preferred instruments' and then only include the SMD computed based on the instrument highest on that list.

    This is another common practice and actually the easiest. So, for example, if instrument X is used, we calculate the SMD based on that; if X is not reported but Y is, then we use that; and if that is not reported, then we use Z; and so on.

    It's important to set up the hierarchy a priori and not let the size of the observed effects guide your selection. Then it's a perfectly defensible approach that avoids all of the complexities described above.

  • $\begingroup$ Thanks, its very helpful. Can I ask you another question? A study in my meta-analysis examined quality of life. They used two instruments in a sample of adolescents and young adults. One instrument for participants up to 18 years and the other instrument for older participants. Can I combine these two? Or can I include both because the samples are somewhat different? $\endgroup$ – Diana Apr 2 '14 at 7:19
  • $\begingroup$ If I understand you correctly, you have two SMDs for this study, one based on a group of adolescents and one based on a group of adults. Since the two samples are independent (i.e., no overlap between the two groups), the two SMDs are also independent. So, you could include them both. However, one may still want to account for the multilevel structure of the data (SMDs nested within studies), which again requires more elaborate models. See, for example, this paper: onlinelibrary.wiley.com/doi/10.1002/jrsm.35/abstract $\endgroup$ – Wolfgang Apr 2 '14 at 16:06
  • $\begingroup$ Yes,thats right. Its an overall sample of n=371; 11 filled out questionnaire A and n=360 answered questionnaire B, both questionnaires assessed quality of life. And I would compute two effect sizes (Hedges'g) and display them in a forest plot. Can I do that? $\endgroup$ – Diana Apr 2 '14 at 19:41
  • $\begingroup$ Yes, but if you plan on conducting an actual meta-analysis with these values, then again I would recommend reading the paper that I linked to. $\endgroup$ – Wolfgang Apr 2 '14 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.