Combining outcome measures for meta-analysis

Question

I have to conduct a meta-analysis. Following problem occured: a single study have measured Quality of Life and therefore, used two questionnaires. Now is my question whether I have two choose one of the instrument to include in meta-analysis or is there a possibility to combine the means of the two scales to provide one standard mean difference in the meta-analysis? THANK YOU!!!!

Answer 1

At least 3 different approaches are possible:

1. You compute two standardized mean differences for this study and include both of these values in the meta-analysis.

   Since the two effects were computed based on the same sample, they are not independent. Therefore, you have to account for and incorporate the degree of dependence between the two estimates in the analysis. This means that you will have to calculate the covariance between the two estimates, which requires knowing the size of the correlation of the raw scores for the two instruments. This is typically not reported, so at that point, you would have to 'guestimate' a reasonable value for the correlation and then conduct a sensitivity analysis in the end (by varying the guestimated value within a reasonable range and then checking whether the conclusions remain unchanged).

   The equation for the covariance can be found in:

   Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York: Russell Sage Foundation.

   The model required for the analysis is also a bit more complex. If you are using R, you could take a look at: http://www.metafor-project.org/doku.php/analyses:gleser2009#multiple-endpoint_studies where I actually describe this type of analysis based on the Gleser & Olkin (2009) chapter (although in practice, one may consider adding random effects to the model to model heterogeneity).

   While this approach maximizes the amount of information extracted from the studies, it comes with a lot of additional complexities that are hardly worth it if you only have one or two studies like that.

2. You compute two standardized mean differences for this study and average them.

   This is in fact a common practice in meta-analysis. From a purely statistical perspective, this approach is not quite right. The average itself is fine, but the sampling variance of the average is typically not computed in the right way. Computing this correctly again requires knowing the correlation of the raw scores. So, while this approach avoids the additional complexities of having to use a model that accommodates multiple effects for the same study, if you want to do this in the correct way, you still need information that is not typically reported and extra steps to check for the sensitivity of the conclusions to the 'guestimated' size of the correlation.

3. You set up a hierarchy of 'preferred instruments' and then only include the SMD computed based on the instrument highest on that list.

   This is another common practice and actually the easiest. So, for example, if instrument X is used, we calculate the SMD based on that; if X is not reported but Y is, then we use that; and if that is not reported, then we use Z; and so on.

   It's important to set up the hierarchy a priori and not let the size of the observed effects guide your selection. Then it's a perfectly defensible approach that avoids all of the complexities described above.