I was trying to understand the different use-cases for differences-in-differences models vs. ANCOVA (post-period = pre-period + experiment_group), for observational data. I came across the below article, which starts with:

"When comparing pretest to posttest changes in non-randomized groups, most researchers are correctly avoiding ANCOVA with posttest as the dependent variable and pretest as the covariate. "


Why should that approach be avoided? I have used that approach in the past to control for regression to the mean effect, in models where we are trying to understand which behaviors and covariates are associated with different changes (improvements/worsening) in clinical outcomes.

Also, if anyone has thoughts on the DiD vs. ANCOVA for large samples, I would love to hear!

  • $\begingroup$ I suspect Lord's paradox is one possible reason against the proposed model. $\endgroup$ – COOLSerdash Nov 19 '20 at 20:10
  • $\begingroup$ What is “pre-period” in your regression? Is it the pre-period mean of your outcome as a covariate? $\endgroup$ – Thomas Bilach Nov 19 '20 at 20:16
  • $\begingroup$ @ThomasBilach It's the measurement before the treatment/intervention (e.g. blood-pressure). post-period are the measurements after treatment. Every participants has two measurements: before and after. $\endgroup$ – COOLSerdash Nov 19 '20 at 20:27

This is explained very clearly in Lüdtke and Robitzsch (2020). I'll briefly summarize their arguments below but the paper is clear and easy to read, and I recommend you read it.

Essentially, the change score approach requires two assumptions to be unbiased: 1) the effect of the unmeasured confounders on the outcome is consistent across time points and 2) the pre-treatment outcome does not cause selection into treatment after conditioning on the unobserved confounders; that is, the pre-treatment outcome is an indicator of the unmeasured confounders but itself is not a confounder.

The ANCOVA approach is unbiased when the pre-treatment outcome completely mediates the relationship between the unmeasured confounder and the outcome. This would occur when selection into treatment depends only on the pre-treatment outcome (and possibly other factors unrelated to the post-treatment outcome).

So, the question is, is the pre-treatment outcome simply a proxy for the unmeasured confounders or itself part of the process of treatment selection? If the answer is some of both (which is likely), neither method will be unbiased alone. Other methods of adjusting for confounding, such as including covariates in the outcome model or preprocessing the data with matching or weighting can increase the possibility of the required assumptions being true. Fundamentally, the assumptions are untestable, but substantive knowledge of the treatment selection process can go a long way.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.