I am conducting a meta-analysis of cognitive-behavioural interventions on self-esteem and I could use some assistance navigating through the statistical procedures.

I have extracted pre-post scores from single group designs and experiment/control scores from RCTs and calculated g for self-esteem rating scales and depression rating scales. Based on the nature of the studies, there are four themes that I would like to explore and I would like to ensure that I use the correct statistical tools and set the correct hypotheses.

The first task is to see if there is a difference between single-day workshops and multi-session therapy over several weeks. The hypothesis is that the former will yield smaller effect sizes than the latter. Second, we understand that single-group designs tend to have larger effect sizes than RCTs and we would like to see if that is the case. Third, we would like to test to see if the three different CBT-based interventions that are used are significantly different. We believe that they will not be. Lastly, we would like to know if the level of self-esteem prior to treatment is a predictor of outcome. It is unclear as to the direction that this might take, though. It could be argued that there is a ceiling effect, which limits the impact of an intervention on people with high self-esteem, or that low self-esteem is enduring enough to resist treatment compared to those with higher self-reported self-esteem.

Would it be as simple as creating dichotomous variables for each of the above and then using a meta-regression analysis of the four variables using R or would some of them be better suited to a Z-test between subgroups? I assume that as we would see in primary studies, multiple Z-tests would lead to a rise in the overall alpha. I am mindful that the third variable (intervention) is an a priori acknowledgment of the null hypothesis and the fourth one might be too vague for inclusion, so I would appreciate any guidance on how to proceed correctly.

  • $\begingroup$ You say that you are going to use regression. Are you aware that standard regression procedures do not give the same result as meta-analysis software? See metafor-project.org/doku.php/tips:regression_with_rma for an explanation. $\endgroup$ – mdewey Jun 5 '16 at 11:47
  • $\begingroup$ Yes, thank you. I just edited it to say meta-regression using R. $\endgroup$ – Dan K Jun 5 '16 at 22:42

Meta-regression certainly seems the way to proceed here. You have the question of whether to fit a model with all your moderators simultaneously (assuming you have enough data points), to fit one at a time, or some combination of these. I would have thought that the distinction between RCT and non-RCT was one which any model should include as it is (a) well known (b) likely to bias any other moderator effect. Of the other three it is up to you. I assume self-esteem is a study level variable and so any inference there will be an ecological one and so possibly less interesting. So to summarise, I would always include RCT v n-n-RCT in any model plus whatever else I was testing and I would consider a full model if there are enough data points.

  • $\begingroup$ That raises the question of what is considered to be enough data points. I have 16 studies. Would it be too optimistic to think that we could draw any reasonable conclusion about so many moderators? $\endgroup$ – Dan K Jun 10 '16 at 21:45
  • $\begingroup$ Yes, I think it would be optimistic. $\endgroup$ – mdewey Jun 11 '16 at 12:59
  • $\begingroup$ Okay. Thank you. That is very helpful. With such a k would it be possible to draw any conclusions about moderators? $\endgroup$ – Dan K Jun 11 '16 at 14:02
  • $\begingroup$ One at a time, and possibly two. It does depend a bit on how many you have in each cell. So if there is only one RCT of single day treatment then your ability to look at them together is going to be a bit limited. $\endgroup$ – mdewey Jun 11 '16 at 15:07
  • $\begingroup$ That makes sense. So we would be looking at all RCTs vs all single groups and then all day vs weeks rather than looking at all four cells. With four dichotomous variables, some of the 16 cells won't be represented at all and others will have very few studies in them. Thanks for your help! $\endgroup$ – Dan K Jun 11 '16 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.