Causal Inference on test scores

Question

I administered a test and wanted to know if the exam scores were influenced by watching videos. The participants were randomly entered into 2 arms. I have one control arm that did not watch videos, and the second arm being the group that did watch videos. I administered a pretest, had them watch the videos, and then take a post-test to the groups, acquiring their scores. After tallying them, I combined the two scores into a data set of scores with the pre-test data first and the post-test data afterward. I ran causalimpact in R on the set. Here are the results that I got.

Posterior tail-area probability p:   0.0111
Posterior prob. of a causal effect:  98.89%

I wanted to know if I implemented causal impact appropriately given that I combined both the pretest and posttest data into one set and split based upon when the pre-test data ended and the post-test data started. I also wanted to know if my assumption of causality would make sense in this instance.

Answer 1

I'm new to causal inference, but this sounds like a straightforward application of linear models.

You're interested in computing

$$ Pr(\mbox{Score} \vert do(\mbox{Videos}) ) $$

By randomizing, you have severed any arrows in your dag from confounders to the treatment. So it should be, in principle, as easy as performing a t-test between the group scores.

However, you might say...

But Demetri, what if the subjects in the experimental group all happen to be poor test takers. I need to adjust for pre-test score.

The randomization addresses this issue. If pre-intervention ability lead you to select which subjects got the intervention, then that would be a different story. In that case, you have classic fork confounding where pretest ability causes treatment and the outcome. Conditioning on pre-test ability would be the right thing to do in that scenario, but is not needed in the one you describe.