I'm analyzing data from a single-group, pre-post design, trying to see if student beliefs change after an intervention. Students are nested in classrooms, and there is a significant amount of classroom-level variation both in post scores (30%) and gain scores (20%), so I'm assuming I should account for this and can't just use a repeated-measures ANOVA. Is it reasonable to model gain scores as an outcome in HLM, controlling for pre scores, and simply look at the significance of the intercept as evidence of growth? I plan to also look at other student-level and classroom-level covariates that might explain variation. I have often heard caution against using gain scores, but in this instance I'm not sure what else I could do. Thank you for any suggestions!
1 Answer
You have a couple of options with two time point data.
Your gain score idea could work. When you create a gain score and then add the time 1 variable to your regression model with your treatment variable, you are evaluating whether there is a difference in treatment vs. control gain scores for students with the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."
You can run a "residualized change score model" instead of a gain score model. In such a model, you are regressing the time 2 score on the time 1 score plus treatment and any other covariates. Then your estimate of the intervention effect is its effect on the time 2 score for students with the same test score value at time 1 (and any other covariates you are adjusting for).
Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.
Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):
Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$
-Rearranged: $Y_2 = a + 1Y_1 + bX + e$
Residualized change: $Y_2 = a + \beta Y_1 + bX + e$
Your proposed gain score: $Y_2-Y_1 = a + bY_1 + bX + e$
The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.
-
$\begingroup$ Hi Erik, thanks so much for your response. This is a really helpful explanation of the difference between the residualized change score model and looking at gain scores. The issue I have is that I don't have a control group, so I can't model the effect of the intervention. Without a control group, does the gain score model that I described make sense? $\endgroup$– JMFKCommented Jul 7, 2020 at 19:09
-
$\begingroup$ There is no right answer here. Since you have no control group (all students were exposed to the intervention), it's hard for me to figure out how you answer whether the intervention had an effect on beliefs. You have to think about which of these models best answers your question. Or you alter your question to get at a different question, such as, do some students change more than others and are such differences at all related to classroom factors. $\endgroup$ Commented Jul 8, 2020 at 21:25