2
$\begingroup$

I am currently analyzing data from an RCT. The primary findings from the RCT have been published and we are now examining the data further. In brief, the RCT examined the effectiveness of a new behavioral treatment compared to another empirically supported treatment and a wait-list control group. The initial study found that both treatments were superior to the wait-list based on end of treatment scores (adjusted for baseline scores) on various outcome variables. The two treatments did not significantly differ from one another over all in terms of most outcomes.

We are now interested in seeing whether there are differences in effectiveness of the two treatments depending on baseline characteristics of the sample. We are only interested in comparing the two treatment conditions and are not interested in the wait-list control condition at this time.

I am planning on running ANCOVAs with interaction variables (treatment condition * various continuous baseline predictors) and examining significant interactions to look at differences in the form of the interaction between these variables and Treatment 1 and Treatment 2 on outcomes.

Is it fine if I drop the data from the wait-list control condition if we are not interested in this condition and there are previously published results attesting to the fact that both treatments are more effective than the WL control? Or does this threaten the statistical design in some way?

I am also aware that my analysis method may not be the most cutting edge in terms of addressing heterogeneity in treatment response, but I am not as confident about performing other methods. However, if you believe something other than an ANCOVA would be appropriate here, I am open to thoughts on this analysis plan as well. Thanks in advance!

$\endgroup$

1 Answer 1

1
$\begingroup$

Generally speaking, comparing two groups based on differences in outcomes makes sense. What gives me pause is the sentence, "We are now interested in seeing whether there are differences in effectiveness of the two treatments depending on baseline characteristics of the sample." If the randomization mechanism in the original experiment worked, then any group differences in the interactions between baseline characteristics and treatment is due to chance.

I'd take a step back at this point and examine the motivation for the research question. What is the rationale to study interactions for just these two treatments in a new study? Why were hypothesized interactions not verified in the original experiment?

$\endgroup$
4
  • $\begingroup$ Sorry, perhaps I was unclear in my wording. We are not interested in group differences on baseline characteristics, but how participant characteristics interact with treatment to predict outcomes. Basically, we are interested in addressing heterogeneity in treatment effects, e.g.: egap.org/resource/… Maybe the confusion was that I stated ANCOVA, but it's more like examining interaction effects in a regression model. $\endgroup$ Commented Aug 10, 2022 at 16:09
  • $\begingroup$ The initial study aims were to examine whether the new treatment, treatment A, functioned better than 1) a WL control and 2) an existing treatment. It was found that it did outperform WL, as did the existing treatment, but that the average treatment effect between treatments A and B didn't differ. $\endgroup$ Commented Aug 10, 2022 at 16:11
  • $\begingroup$ Now I'm curious about heterogeneity in treatment effects between treatment A and B. If I were to examine the interaction between treatment*baseline characteristics in a regression model in predicting outcomes, would it be ok to just have treatment have two levels and drop the third level of data from the WL condition? $\endgroup$ Commented Aug 10, 2022 at 16:11
  • 1
    $\begingroup$ Thanks for the clarification. You can drop control and estimate the regression of y on (treatment * baseline) and comment on differences to motivate a future experiment, but you can't make causal statements from this one. I'm assuming that the original experiment did not follow a factorial design, which is what you'd need to study interaction effects. $\endgroup$
    – wahid
    Commented Aug 10, 2022 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.