0
$\begingroup$

I am trying to evaluate the effect of a certain policy on the achievement gains of students between two points in time. However, not all the students are exposed to this policy. I am supposed to use the gain score as a dependent variable.

After having read many critical articles about the use of gain scores as outcome variables, I would like to ask if somebody could tell me the advantages of this approach (compared to using the post-test score as an outcome variable while controlling for the pre-test score).

Is it wrong to say that only when using the difference score as an outcome I can really answer the question of whether competences of students grow faster when they are exposed to this policy? I feel like using the post-test score as an outcome variable does not precisely give me an answer to this question.

I would really appreciate if somebody could give me some advice on that

$\endgroup$
1

1 Answer 1

1
$\begingroup$

The issue you are describing is well understood in the context of clinical trials, see for example Chapter 7.2.3 in Senn (2008).

You may think that comparing difference scores (also known as change scores) and comparing post-test scores while adjusting for pre-test scores (also known as analysis of covariance, ANCOVA) are different, however, the former is actually a special case of the latter.

To see this, denote by $Y_{1i}$ and $Y_{2i}$ the pre- and post-test score of student $i$, respectively. Whether student $i$ received the policy is coded with a dummy variable $X_i$, which is 1 if they received it and 0 otherwise. The ANCOVA model is now given by $$ Y_{2i} = \mu + \beta X_i + \gamma Y_{1i} + \epsilon_i, $$ where $\mu$ is the average score when receiving no policy, $\beta$ the effect of the policy, $\gamma$ the influence of the pre-test score on the post-test score, and $\epsilon_i$ a random error.

The change-score model, on the other hand, is given by $$ Y_{2i} - Y_{1i} = \mu + \beta X_i + \epsilon_i, $$ which can be identified as a special case of the ANCOVA model with $\gamma = 1$ and $Y_{1i}$ subtracted from both sides. In contrast, if we set $\gamma = 0$, we obtain a model where the post-test score does not depend on the pre-test score at all, the reality is probably somewhere in between.

What this means in practice is that the change-score model is a more restrictive model compared to the ANCOVA model. It may well be the case that the influence of the pre-test score is not equal to one, which in turn will reduce the efficiency of your estimate of $\beta$.

$\endgroup$
3
  • $\begingroup$ Thank you so much for your answer ! That really helps a lot. $\endgroup$ Jan 31, 2021 at 20:50
  • $\begingroup$ But still, is there any advantage of the difference score over the "normal" ANCOVA model ? The supervisor of my thesis wants me to use the difference score variant and I don't really understand why $\endgroup$ Jan 31, 2021 at 20:51
  • $\begingroup$ Senn recommends to choose ANCOVA over change-score analysis as the default approach. However, there are some deeper issues to be aware of, the famous Lord's paradox, see for example this blog post by Senn or this answer which compiled a nice list of references. $\endgroup$ Feb 1, 2021 at 19:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.