# Extreme differences between control group and affected group

I am reviewing an article and cannot be overly specific but it involves one group of people with a medical condition and another group without it; the dependent variables are various mental health issues. The problem is that the two groups are radically different on a number of relevant demographics (age, sex, race/ethnicity etc. These are related to both the condition and the outcomes). The authors tried to deal with this by adding them as covariates, but my intuition is that this is likely to fail for very large difference. I have a vague memory of reading about this as well, but I can't remember where I read it.

Any thoughts or links appreciated.

EDIT: The sample sizes were 178 cases and 270 controls. The correlation among the covariates wasn't reported, but it can be assumed to be pretty high for some pairs (e.g. ethnic group and rural vs. urban and income).

EDIT 2: What I want to know is a) Whether the extreme differences between the groups make it impossible to properly control for them using regression and b) If so, what else could be done. I'm thinking about matching of various kinds.

• What is the sample size of each group like? How common are the covariates? Are the covariates strongly correlated? Jul 5, 2021 at 16:46
• Hi Alexis, I edited my question. Jul 5, 2021 at 16:52
• The phrase Lord's paradox comes to mind. Perhaps that is what you had stashed somewhere in your cortex? Jul 5, 2021 at 16:58
• Thanks mdewey, but I don't think Lord's Paradox is it. This was a cross-sectional study, and Lord's paradox seems to relate only to data with two time points (or possibly more). Jul 5, 2021 at 17:18
• I have never heard of Lord's paradox! Thank you, @mdewey Jul 5, 2021 at 18:16

There is the potential for bias due to model misspecification when using regression to adjust for confounding. Regression can work, but when the groups are dissimilar to each other, it relies heavily on extrapolation. Regression essentially estimates the missing potential outcomes from the units in the other group. If the groups differ from each other, then the regression model requires extrapolation, which often yields bias. One can rely on extrapolation, but skeptical readers should not accept an extrapolated estimate as being meaningful or useful. This is the central thesis of Ho, Imai, King, and Stuart (2007) and King and Zeng (2006). Rubin also wrote about this as long ago as 1973 in his early works on the analysis of observational studies (Rubin, 1973). You as a reviewer can say that you don't believe the results because they rely heavily on extrapolation and the regression model is subject to misspecification.

The authors should be clear about what estimand they are targeting. I describe how to make this choice in this article (still on arxiv but submitted). A simple regression of the outcome on the treatment and covariates with no interactions between them does not estimate the ATE. The ATE may be impossible to estimate without extrapolation. Instead, the authors might target the ATO, the treatment effect in the population of overlap between the groups. There are methods that directly estimate the ATO with as good precision as possible, such as overlap weighting (Li, Thomas, & Li, 2018). If these are not successful at eliminating imbalance, then the groups are too fundamentally different from each other and in principal incomparable without extrapolation.

Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2007). Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis, 15(3), 199–236. https://doi.org/10.1093/pan/mpl013

King, G., & Zeng, L. (2006). The dangers of extreme counterfactuals. Political Analysis, 14(2), 131–159. https://doi.org/10.1093/pan/mpj004

Li, F., Thomas, L. E., & Li, F. (2018). Addressing Extreme Propensity Scores via the Overlap Weights. American Journal of Epidemiology, 188(1), 250–257. https://doi.org/10.1093/aje/kwy201

Rubin, D. B. (1973). The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies. Biometrics, 29(1), 185–203. https://doi.org/10.2307/2529685