# Causal Inference on test scores

I administered a test and wanted to know if the exam scores were influenced by watching videos. The participants were randomly entered into 2 arms. I have one control arm that did not watch videos, and the second arm being the group that did watch videos. I administered a pretest, had them watch the videos, and then take a post-test to the groups, acquiring their scores. After tallying them, I combined the two scores into a data set of scores with the pre-test data first and the post-test data afterward. I ran causalimpact in R on the set. Here are the results that I got.

Posterior tail-area probability p:   0.0111
Posterior prob. of a causal effect:  98.89%


I wanted to know if I implemented causal impact appropriately given that I combined both the pretest and posttest data into one set and split based upon when the pre-test data ended and the post-test data started. I also wanted to know if my assumption of causality would make sense in this instance.

• The causalimpact library seems like overkill for this design. If I have understood correctly, you randomized participants and then measured pre-test and post-test scores? Dec 3, 2020 at 14:37
• Casual? I guess you mean causal? Care to correct? Dec 3, 2020 at 15:01
• @kjetilbhalvorsen Yep, I will correct, sorry for that. Dec 3, 2020 at 21:06
• @DemetriPananos Yes, I do. Dec 3, 2020 at 21:06
• @user1449249 You definitely don't need causalimpact then. Dec 3, 2020 at 21:37

I'm new to causal inference, but this sounds like a straightforward application of linear models.

You're interested in computing

$$Pr(\mbox{Score} \vert do(\mbox{Videos}) )$$

By randomizing, you have severed any arrows in your dag from confounders to the treatment. So it should be, in principle, as easy as performing a t-test between the group scores.

However, you might say...

But Demetri, what if the subjects in the experimental group all happen to be poor test takers. I need to adjust for pre-test score.

The randomization addresses this issue. If pre-intervention ability lead you to select which subjects got the intervention, then that would be a different story. In that case, you have classic fork confounding where pretest ability causes treatment and the outcome. Conditioning on pre-test ability would be the right thing to do in that scenario, but is not needed in the one you describe.

• A t-test would be fine, but an ANCOVA controlling for pre-test would make the test way more powerful and the estimate of the effect more precise even with randomization. One should always adjust for pre-treatment predictors of the outcome (including pre-tests scores) when estimating effects after randomized trials.
– Noah
Dec 3, 2020 at 22:41
• I will accept this answer as it demonstrates that I didn't need a method of statistical analysis I thought I did. Thanks @DemtriPananos Dec 4, 2020 at 1:00