Why is ANCOVA not appropriate for modelling post-intervention outcome, controlling for baseline

Question

I was trying to understand the different use-cases for differences-in-differences models vs. ANCOVA (post-period = pre-period + experiment_group), for observational data. I came across the below article, which starts with:

"When comparing pretest to posttest changes in non-randomized groups, most researchers are correctly avoiding ANCOVA with posttest as the dependent variable and pretest as the covariate. "

https://www.sciencedirect.com/science/article/abs/pii/S0167876003002757

Why should that approach be avoided? I have used that approach in the past to control for regression to the mean effect, in models where we are trying to understand which behaviors and covariates are associated with different changes (improvements/worsening) in clinical outcomes.

Also, if anyone has thoughts on the DiD vs. ANCOVA for large samples, I would love to hear!

Answer 1

This is explained very clearly in Lüdtke and Robitzsch (2020). I'll briefly summarize their arguments below but the paper is clear and easy to read, and I recommend you read it.

Essentially, the change score approach requires two assumptions to be unbiased: 1) the effect of the unmeasured confounders on the outcome is consistent across time points and 2) the pre-treatment outcome does not cause selection into treatment after conditioning on the unobserved confounders; that is, the pre-treatment outcome is an indicator of the unmeasured confounders but itself is not a confounder.

The ANCOVA approach is unbiased when the pre-treatment outcome completely mediates the relationship between the unmeasured confounder and the outcome. This would occur when selection into treatment depends only on the pre-treatment outcome (and possibly other factors unrelated to the post-treatment outcome).

So, the question is, is the pre-treatment outcome simply a proxy for the unmeasured confounders or itself part of the process of treatment selection? If the answer is some of both (which is likely), neither method will be unbiased alone. Other methods of adjusting for confounding, such as including covariates in the outcome model or preprocessing the data with matching or weighting can increase the possibility of the required assumptions being true. Fundamentally, the assumptions are untestable, but substantive knowledge of the treatment selection process can go a long way.

Answer 2

DiD vs. ANCOVA has been discussed in my workplace, a health insurance company, where DiD has been the industry standard for some time. I agree that DiD does not adequately account for regression to the mean effects.

A simplified causal diagram for a typical pre/post study I've worked on is illustrated below:

where: A = the health intervention (treatment),

Y0 = the pre-treatment outcome (summed over a given time period, say 12 months),

Y1 = the post-intervention outcome,

Y1 - Y0 = the change score outcome for a DiD study.

In this particular study, Y0 is a confirmed confounder of both A and Y1 and A and Y1-Y0. There is an uncertain relationship between Y1 and Y1-Y0 -- as Tennant et al. have pointed out, it's not obvious whether Y1 causes Y1-Y0 or Y1-Y0 causes Y1.

Based on this diagram it should be clear that Y0 is a confounder which ought to be conditioned on when estimating either the average treatment effect of A on Y1-Y0 in a DiD model, or A on Y1 in the ANCOVA. In addition, conditioning on Y0 controls for regression to the mean effects.

Note that both the DiD regression and simplified ANCOVA:

$E[Y_1-Y_0]=\alpha_0 + \alpha_1Y_0 + \alpha_2A$, and

$E[Y_1]=\beta_0 + \beta_1Y_0 + \beta_2A$

produce the same estimates of the treatment effect ($\alpha_2=\beta_2$). Hence, in this limited case with two time periods DiD doesn't appear to be necessary.