3
$\begingroup$

There have been a number of papers over the past 40 years supporting either change score or regressor analysis for longitudinal observational studies.

However, recent papers (for example, Tennant et al. (2022) and Shahar & Shahar (2010)) argue change scores are not meaningful quantities and not suitable for causal inference of observational data.

Without getting into the technical details of these articles, it's not difficult to create simple case-control simulations where regressing a $Y_1-Y_0$ change score on a binary treatment/control group variable $X$ produces (seemingly) misleading results compared with a standard regression of $Y_1$ on $Y_0$ and $X$.

In both examples below, what appears to be a significant treatment effect in the first data set, and a non-significant treatment effect in the second data set are completely reversed in the change score analysis.

## Significant treatment effect (X=1) for regressor method, not significant for change score method
    set.seed(123)
    df_trmt <- data.frame(x=c(rep(1,100)), y1 = c(rnorm(100, 20, 8)))
    df_trmt$y2 = c(rnorm(100, 15 + .25*df_trmt$y1, 2)) 
    df_ctrl <- data.frame(x=c(rep(0,100)), y1 = c(rnorm(100, 40, 8)))
    df_ctrl$y2 = c(rnorm(100, 30 + .25*df_ctrl$y1, 2))
    df <- rbind(df_trmt, df_ctrl)
    df$z = as.factor(df$x)
    
    plot(df$y1, df$y2, col=c("black","gray50")[df$z], xlim=c(0,100), ylim=c(0,100))
    a = seq(0, 100, 1)
    b = seq(0, 100, 1)
    lines(a, b, col="blue")
    
    df$diff = df$y2 - df$y1
    boxplot(df$diff ~ df$x)
    
    # Regressor method 
    summary(lm(df$y2 ~ df$y1 + df$x))
    
    # Change score method
    summary(lm(df$diff ~ df$x))

## No significant treatment effect (X=1) for regressor method, significant for change score method
    set.seed(234)
    df_trmt <- data.frame(x=c(rep(1,100)), y1 = c(rnorm(100, 20, 6)))
    df_trmt$y2 = c(rnorm(100, 30 + .25*df_trmt$y1, 2)) 
    df_ctrl <- data.frame(x=c(rep(0,100)), y1 = c(rnorm(100, 60, 6)))
    df_ctrl$y2 = c(rnorm(100, 30 + .25*df_ctrl$y1, 2))
    df <- rbind(df_trmt, df_ctrl)
    df$z = as.factor(df$x)
    plot(df$y1, df$y2, col=c("black","gray50")[df$z], xlim=c(0,100), ylim=c(0,100))
    a = seq(0, 100, 1)
    b = seq(0, 100, 1)
    lines(a, b, col="blue")

    # Regressor method 
    summary(lm(df$y2 ~ df$y1 + df$x))
    
    # Change score method
    summary(lm(df$diff ~ df$x))

Can we finally resolve the debate in favor of the regressor method?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ The simplest heuristic argument in favor of the ANCOVA approach is that it is a generalization of a change score analysis. That is IF the change score is right, you simply estime the baseline (y0) coefficient as 1. For they and other reasons I'm in the "it's dead" camp. $\endgroup$
    – AdamO
    Jan 7 at 5:24
  • $\begingroup$ Case-control designs are weak sauce for causal inference, so not sure where your argument is coming from there. Also: without measuring some kind of change score (DID, RCT, cohort design, etc.) it is not clear how you can measure change which is necessary for causal inference. $\endgroup$
    – Alexis
    Jan 7 at 5:24
  • $\begingroup$ Difference in difference, interestingly, is a whole other topic. You can conduct a difference in difference analysis with the untransformed, raw response as the outcome. The language around what, exactly, DiD is is weak, but it is most easily understood as an interaction. $\endgroup$
    – AdamO
    Jan 7 at 5:25
  • 1
    $\begingroup$ @Alexis As I understand it, the position in the Tennant paper is that a change score is not a random variable. Instead it's a variable derived from Y0 and Y1 & therefore contains no new information. In causal diagrams for DiD models does it make sense to include Y0, Y1, and Y1-Y0? I'm leaning in the No direction. There's also the issue of change scores producing contradictory results (see my examples). $\endgroup$
    – RobertF
    Jan 7 at 15:15
  • 2
    $\begingroup$ You might be interested in my question and answer: How to represent difference variables in DAGs. $\endgroup$
    – Alexis
    Jan 7 at 18:59

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy