Skip to main content
Cross Validated

Return to Answer

Edited equations
Source Link
Erik Ruzek
Erik Ruzek
  • 5.9k
  • 13
  • 22

You have a couple of options with two time point data.

  1. Your gain score idea could work. When you create a gain score and then add the time 1 variable to your regression model with your treatment variable, you are evaluating whether there is a difference in treatment vs. control gain scores for students with the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."

  2. You can run a "residualized change score model" instead of a gain score model. In such a model, you are regressing the time 2 score on the time 1 score plus treatment and any other covariates. Then your estimate of the intervention effect is its effect on the time 2 score for students with the same test score value at time 1 (and any other covariates you are adjusting for).

Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.

Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):

Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$

-Rearranged: $Y_2 = a + 1Y_1 + bX + e$

Residualized change: $Y_2 = a + \beta Y_1 + bX + e$

Your proposed gain score: $Y_2-Y_1 = a + \beta Y_1 + bX + e$$Y_2-Y_1 = a + bY_1 + bX + e$

The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.

You have a couple of options with two time point data.

  1. Your gain score idea could work. When you create a gain score and then add the time 1 variable to your regression model with your treatment variable, you are evaluating whether there is a difference in treatment vs. control gain scores for students with the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."

  2. You can run a "residualized change score model" instead of a gain score model. In such a model, you are regressing the time 2 score on the time 1 score plus treatment and any other covariates. Then your estimate of the intervention effect is its effect on the time 2 score for students with the same test score value at time 1 (and any other covariates you are adjusting for).

Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.

Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):

Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$

-Rearranged: $Y_2 = a + 1Y_1 + bX + e$

Residualized change: $Y_2 = a + \beta Y_1 + bX + e$

Your proposed gain score: $Y_2-Y_1 = a + \beta Y_1 + bX + e$

The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.

You have a couple of options with two time point data.

  1. Your gain score idea could work. When you create a gain score and then add the time 1 variable to your regression model with your treatment variable, you are evaluating whether there is a difference in treatment vs. control gain scores for students with the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."

  2. You can run a "residualized change score model" instead of a gain score model. In such a model, you are regressing the time 2 score on the time 1 score plus treatment and any other covariates. Then your estimate of the intervention effect is its effect on the time 2 score for students with the same test score value at time 1 (and any other covariates you are adjusting for).

Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.

Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):

Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$

-Rearranged: $Y_2 = a + 1Y_1 + bX + e$

Residualized change: $Y_2 = a + \beta Y_1 + bX + e$

Your proposed gain score: $Y_2-Y_1 = a + bY_1 + bX + e$

The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.

Source Link
Erik Ruzek
Erik Ruzek
  • 5.9k
  • 13
  • 22

You have a couple of options with two time point data.

  1. Your gain score idea could work. When you create a gain score and then add the time 1 variable to your regression model with your treatment variable, you are evaluating whether there is a difference in treatment vs. control gain scores for students with the same time 1 value on the outcome. The intercept in this model is the gain score for a student in the control condition and a value of 0 on the time 1 test score. I don't view that as "evidence of growth."

  2. You can run a "residualized change score model" instead of a gain score model. In such a model, you are regressing the time 2 score on the time 1 score plus treatment and any other covariates. Then your estimate of the intervention effect is its effect on the time 2 score for students with the same test score value at time 1 (and any other covariates you are adjusting for).

Both are valid, but the change score model has an additional assumption baked in (see below). Likewise, the second approach can increase power to detect the effect of interest because time 1 is typically strongly correlated with the time 2 outcome. It is unclear to me whether your proposal to additionally control for time 1 score in your gain score model will help you power-wise. It might if those who are higher at time 1 typically change less (or more) from time 1 to time 2, but wouldn't help you if this was not the case.

Ignoring the multilevel aspect and assuming $X$ is treatment, note the difference between the models (hat tip to David Kenny):

Traditional gain score (no time 1 control): $Y_2-Y_1 = a + bX + e$

-Rearranged: $Y_2 = a + 1Y_1 + bX + e$

Residualized change: $Y_2 = a + \beta Y_1 + bX + e$

Your proposed gain score: $Y_2-Y_1 = a + \beta Y_1 + bX + e$

The residualized change score model empirically estimates the $\beta$ for $X$ while the traditional gain score model fixes this $\beta$ at 1. Your proposed gain score model still fixes the $\beta$ at 1, but further allows for gains to be linearly associated with the time 1 value. I'm not sure this is needed, unless your theory is specifically about this issue. Most educational research I see (and this is my field) uses a residualized change score model in these situations.