Modeling differences in test scores after intervention

Question

I'm trying to understand how to either adjust my data, model it differently, or both.

I have an experiment where the subject is given a test, then an Intervention (A or B), then given another test. I want to see what the main effect of the intervention is on the difference between pre and post intervention scores.

The data

My dataset looks like this:

Subject Session Condition Intervention Difficulty Score
1       1       Pre       A            1          80
1       1       Post      A            2          70
1       2       Pre       B            2          75
1       2       Post      B            2          80
.
.
.
50      9       Pre       A            5          80
50      9       Post      A            6          70
50      10      Pre       A            6          65
50      10      Post      A            5          80

What I tried

Originally, I set up my model as repeated measures ANOVA like this:

modelA <- aov(dScore ~ Intervention + Error(Subject/Intervention))

Where dScore is the difference in Score between the Pre and Post intervention for a given Session.

Problems

I have two problems with this model:

First, the testing protocol during the study dictates that when a subject performs particularly well, the difficulty of the test is increased. Similarly when the subject performs poorly, the difficulty is decreased. You can see how that might look in the dummy data above.

To this end, my ideas are to either apply a z-score within difficulty levels - creating ZScore, or add Difficulty as a covariate.

Second, there are some subjects who received both interventions during the course of the study, and others that only received one or the other. This currently isn't being accounted for in the model, as far as I can tell.

My ideas for solutions

The idea I have is to use a mixed effects model instead of a repeated measures ANOVA. Something like this:

modelB <- lme(dScore ~ Intervention + Difficulty, random = ~ 1 + Intervention | Subject)

Or, splitting Condition back out but using ZScore instead of dScore:

modelC <- lme(ZScore ~ Condition*Intervention, random = ~ 1 + Condition | Subject)

Can someone explain to me whether these are sensible approaches, or what I can be doing differently?

Answer 1

Yes, I think that in general what you propose appears rather reasonable.

Both the idea of using a linear mixed effects model as well as using information about the administered test's Difficulty seem like reasonable solutions to modelling the effect of this intervention. As mentioned I do not see the reason to immediately z-transform your response variable; I would suggest that only if one wanted to change the way the $\beta$ coefficients are interpreted and/or there were some numerical issues during model estimation.

Based on the available information I think that a model similar to: Score ~ Condition*Difficulty + Intervention (1 |Condition:Subject) (I use lmer syntax) seems more reasonable. (Condition|Subject) specifies that the effects of Condition vary across levels of Subject; this is probably too pessimistic and a bit over-parametrized too (this what is usually referred as randomized block design by the way). Taking this, one notch down is: (1|Condition/Subject). This specification would be equivalent to (1|Condition) + (1|Condition:Subject); it would mean one estimates a random effect for Condition based on only two levels which is just wrong so again I would not suggest it (this what is usually referred as nested design). This brings us to (1|Condition:Subject). I think this is plausible: one would fit an intercept for the interaction between Condition and Subject.

Something that worries me is that you do not account for Session explicitly. Are all sessions different and/or independent from the others? Usually some "learning" takes place between sessions. One would normally suspect that the subjects (if anything) would get more accustomed to the testing procedure. I think that it would be sensible to at least test Session as a fixed effect. To that extent, one could try to have random slopes due to Session within Subject with something like (Session+0|Subject) but this is a bit more open-ended. (Notice that this syntax does not allow variation based on a Subject intercept. I proposed it because I assume that you already have the Condition:Subject terms.) Feel free to shoot-down this worry! :)

General advice: As your design is non-trivial I would suggest you try to generate some data based on the assumptions you are willing to make and on the constants your original modelling task has (sample sizes, number of session per subject, number of subjects, etc.). Then you could try to fit the model you propose on this data and see if you get back reasonable estimates. Testing different random effect structures becomes a cherry-picking exercise trying to "squeeze more significance out of the remaining terms" (glmm.wikidot's FAQ) after a while so I would advice you against it. And bootstrap that model in the end. Unless you have parameters located close to the parameter boundary space this usually is a very good test.

Answer 2

Well normally one would do a paired t-test on the difference between the score on the pre- and post-test.

The problem here is that your setup isn't very clean from a theoretical perspective. Three issues:

• You mention that some participants got both treatments. This could be solved by calling this a third group.
• If you want to know what the effect of a treatment is, you also need a no-treatment group, as a baseline. But maybe you are only interested in the differences between these two treatments.
• The difficulty scales could be a show stopper here. Because it is unclear what the effect is of a level change is on the score. Trying to model this difference would only make sense if you have a very large dataset. I'm guessing it would be better to just repeat the experiment with the difficulty locked between the two sessions.