I'm trying to get my head around which effect size is most meaningful for a 2x2 mixed ANOVA for two intervention groups. I have pre and post assessement data (measured at the continuous level) for two different intervention groups (A and B).

For ease of asking this question I've made the image below as a reference and made the mean and standard deviation straight forward just for this example as I'm confused about whether an effect size calculation is needed for pre-to-post within groups analyses (analyses 1 and 2 below), pre/post between groups analyses (analyses 3 and 4 below) or an analysis of the mean difference in pre-to-post assessment scores between the groups (analysis 5 below).

I'm guessing that I should be doing a between groups t-test to demonstrate that the two groups did not differ at pre-assessment (analysis 3 below) and then a t-test on the mean difference in pre-to-post scores between groups (analysis 5 below)?

Edit: Participants were allowed to choose which intervention group they attended, so participants were not randomly assigned to groups.

enter image description here

  • $\begingroup$ Where the interventions assigned by randomization? This makes a huge difference for the answer. $\endgroup$ – Björn May 24 '16 at 5:21
  • $\begingroup$ Participants (clients) were allowed to choose which intervention they wanted so it wasn't random assignment. $\endgroup$ – SwingingStrawberry May 24 '16 at 7:19

If this had been a study with randomized assignment of participants to A or B, then an analysis of covariance for the post-intervention score (or equivalently for the pre-intervention to post-intervention change) with a factor for intervention and a covariate for the pre-intervention score would be one logical approach (assuming this is a approx. continuous score with residuals that could plausibly look approx. normal). Within group pre-/post-tests have very little value for establishing for comparing A versus B and the within group results are mostly of use for describing what happened in terms of what the changes were (not for testing any hypotheses). One can of course compare end of intervention values (or changes to the end of the intervention), but this typically needlessly ignores the usually very informative information contained in the pre-intervention covariate.

Since interventions were not assignment by randomization, confounding could be a major issue. A non-significant test for whether the pre-intervention scores differ between intervention groups is insufficient to establish that this is no the case (in fact a failure to reject a null hypothesis does not establish that it is appropriate to proceed as if the null hypothesis were true). One way of dealing with the potential confounding is a propensity score analysis (e.g. stratifying an analysis of covariance as described above by deciles of the propensity score) with a propensity score for being on intervention A or B developed using e.g. logistic regression including all variables that might explain why participants end up in group A or B (especially if these might also influence the final outcome). It is crucial that all such variables must be taken into account and if not all of them are available, this is a major limitation of the analysis, one should consider trying to obtain the unavailable data (if possible), it should be prominently highlighted and conclusions should be appropriately cautious (e.g. if there appears to be an effect one might conclude that "the studies suggest that there might be an effect of the intervention, but the study results could also be explained by unmeasured confounding variables"), and the nature of the confounding variables and their potential influence should be discussed (e.g. if the unavailable potential confounders are considered to be relatively unimportant, it may be that an extremely large observed effect is unlikely to be explained by confounders).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.