Gain scores as an outcome in HLM? I'm analyzing data from a single-group, pre-post design, trying to see if student beliefs change after an intervention. Students are nested in classrooms, and there is a significant amount of classroom-level variation both in post scores (30%) and gain scores (20%), so I'm assuming I should account for this and can't just use a repeated-measures ANOVA. Is it reasonable to model gain scores as an outcome in HLM, controlling for pre scores, and simply look at the significance of the intercept as evidence of growth? I plan to also look at other student-level and classroom-level covariates that might explain variation. I have often heard caution against using gain scores, but in this instance I'm not sure what else I could do. Thank you for any suggestions!
 A: You have a couple of options with two time point data.

*

*Your gain score idea could work. When you create a gain score and
then add the time 1 variable to your regression model with your
treatment variable, you are evaluating whether there is
a difference in treatment vs. control gain scores for students with
the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."


*You can run a "residualized change score model" instead of a gain
score model. In such a model, you are regressing the time 2 score on
the time 1 score plus treatment and any other covariates. Then your
estimate of the intervention effect is its effect on the time 2
score for students with the same test score value at time 1 (and any
other covariates you are adjusting for).
Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.
Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):
Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$
-Rearranged: $Y_2 = a + 1Y_1 + bX + e$
Residualized change: $Y_2 = a + \beta Y_1 + bX + e$
Your proposed gain score: $Y_2-Y_1 = a + bY_1 + bX + e$
The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.
