1
$\begingroup$

I administered a test and wanted to know if the exam scores were influenced by watching videos. The participants were randomly entered into 2 arms. I have one control arm that did not watch videos, and the second arm being the group that did watch videos. I administered a pretest, had them watch the videos, and then take a post-test to the groups, acquiring their scores. After tallying them, I combined the two scores into a data set of scores with the pre-test data first and the post-test data afterward. I ran causalimpact in R on the set. Here are the results that I got.

Posterior tail-area probability p:   0.0111
Posterior prob. of a causal effect:  98.89%

I wanted to know if I implemented causal impact appropriately given that I combined both the pretest and posttest data into one set and split based upon when the pre-test data ended and the post-test data started. I also wanted to know if my assumption of causality would make sense in this instance.

$\endgroup$
6
  • 2
    $\begingroup$ The causalimpact library seems like overkill for this design. If I have understood correctly, you randomized participants and then measured pre-test and post-test scores? $\endgroup$ Dec 3, 2020 at 14:37
  • $\begingroup$ Casual? I guess you mean causal? Care to correct? $\endgroup$ Dec 3, 2020 at 15:01
  • $\begingroup$ @kjetilbhalvorsen Yep, I will correct, sorry for that. $\endgroup$ Dec 3, 2020 at 21:06
  • $\begingroup$ @DemetriPananos Yes, I do. $\endgroup$ Dec 3, 2020 at 21:06
  • 1
    $\begingroup$ @user1449249 You definitely don't need causalimpact then. $\endgroup$ Dec 3, 2020 at 21:37

1 Answer 1

2
$\begingroup$

I'm new to causal inference, but this sounds like a straightforward application of linear models.

You're interested in computing

$$ Pr(\mbox{Score} \vert do(\mbox{Videos}) ) $$

By randomizing, you have severed any arrows in your dag from confounders to the treatment. So it should be, in principle, as easy as performing a t-test between the group scores.

However, you might say...

But Demetri, what if the subjects in the experimental group all happen to be poor test takers. I need to adjust for pre-test score.

The randomization addresses this issue. If pre-intervention ability lead you to select which subjects got the intervention, then that would be a different story. In that case, you have classic fork confounding where pretest ability causes treatment and the outcome. Conditioning on pre-test ability would be the right thing to do in that scenario, but is not needed in the one you describe.

$\endgroup$
2
  • $\begingroup$ A t-test would be fine, but an ANCOVA controlling for pre-test would make the test way more powerful and the estimate of the effect more precise even with randomization. One should always adjust for pre-treatment predictors of the outcome (including pre-tests scores) when estimating effects after randomized trials. $\endgroup$
    – Noah
    Dec 3, 2020 at 22:41
  • $\begingroup$ I will accept this answer as it demonstrates that I didn't need a method of statistical analysis I thought I did. Thanks @DemtriPananos $\endgroup$ Dec 4, 2020 at 1:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.