3
$\begingroup$

Question, brief

In a single multi-level meta-analysis, is it appropriate to include subscale scores (which represent part of a measure) in addition to total scale scores (representing the full measure), all nested under study?

Question, detailed

I am conducting a meta-analysis on the correlation between two constructs. One of these constructs is frequently measured using a questionnaire that consists of two subscales.

To make this clearer, let's say Measure A is my measure of Construct 1. Measure A contains Subscale 1 and Subscale 2. For some studies, I have the correlation between the total score, as well as both subscale scores, and Measure B (which measures Construct 2). For some studies, I only have the correlation between Measure A (total) and Measure B. For some studies, I only have the correlation between a subscale score and Measure B.

Measurement of Construct 1 in each study (I have the correlation of each of these scores with my measure of Construct 2 (Measure B)):

  • Study 1: MeasureA-Total, MeasureA-Subscale1, MeasureA-Subscale2
  • Study 2: MeasureA-Total
  • Study 3: MeasureA-Subscale1, MeasureA-Subscale2
  • Study 4: MeasureA-Subscale1
  • Study 5: MeasureA-Subscale2
  • Study 6: MeasureA-Total, MeasureA-Subscale1
  • Study 7: MeasureA-Total, MeasureA-Subscale2

Would it be appropriate, statistically speaking, in the case of study 1, to include the correlation of each form of Measure A with Measure B in my analysis (three different correlations)? Because Measure A, Subscale 1 makes up part of the Measure A total score, there is dependency between these scores. However, it seems to me that my multi-level analysis should account for this dependency.

I would like to include both the correlations involving the total score and the subscale scores to maximize the number of effect sizes I am including in my analysis. I would like to test each subscale as a moderator, and if I do not include the effect sizes involving the subscales scores in my initial analysis, this test of moderation would not include as many effects as it might otherwise.

The other option I can see would be for me to use the total scale score, when available, and use the subscale score(s) when the total scale score is not available.

Background

I have consulted the literature but have not found this case specifically addressed.

However, on the topic of dependency in data, Van den Noortgate et al. (2013) write,

"multilevel models automatically account for the hierarchical structure in the data. If, for instance, one study results in 20 effect size estimates, this study will not contribute 20 times as much to the estimation of the mean effect, as compared with a study reporting only 1 effect size. Rather, this study is regarded as only one study yielding information about one study-specific mean effect in the distribution of study mean effects. The exact weight of each study will depend on the dependence between effect sizes from the same study: The smaller this dependence, the less the weight given to each of the individual effect sizes depends on the number of effect sizes reported in the study."

Cited

Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594. https://doi.org/10.3758/s13428-012-0261-6

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.