# Treatment Effect Insignificant with OLS but Significant After IPTW

I'm working with some observational data and wanted to assess the effect of variable (D) on outcome (Y) after controlling for a vector of covariates (X). As the data is observational, I wanted account for as much selection as possible, so I employed (augmented) inverse probability of treatment weighting. After doing so, I found a significant effect of D on Y.

However, I was curious and compared the results to a standard OLS linear regression, which included the vector of covariates. The results for this approach provided evidence of a statistically insignificant effect of D on Y.

If the scenario were reverse and I had found a significant effect for the standard OLS approach but an insignfiicant effect after IPTW, then it would suggest the effect is driven by selection. However, I am confused about how to make sense of my situation where the standard OLS results are insignificant but the IPTW results are significant.

Is this an example of a suppression effect? I'm not sure how to make sense of this.

• Are the CIs overlapping by a lot? Aug 3, 2023 at 23:35
• Hi @dimitriy, yes the CI from the OLS approach includes the coefficient from the AIPW and its lower bound CI Aug 4, 2023 at 3:24
• That suggests that they are not statistically distinguishable from each other. Aug 4, 2023 at 3:46

## 1 Answer

There are a few problems with your inference. Failing to find a statistically significant effect is not the same as "provide[ing] evidence of a statistically insignificant effect". A nonsignificant effect simply means that the null hypothesis is inconclusive, not that it is true.

Second, you claim "if I had found a significant effect for the standard OLS approach but an insignfiicant effect after IPTW, then it would suggest the effect is driven by selection." This doesn't make sense. It means the same thing as the current situation: one of the estimates may be biased and the other may not be. The problem is that you don't know which is which. Confounding ("selection") doesn't always attenuate observed effects; it can amplify them or amplify them in the wrong direction.

So the question comes down to which estimate you should trust more. We don't have enough details to be able to assess which models you fit (OLS is an estimation procedure, not a model, and AIPW is an estimator that requires two models, neither of you which you specified).

In general, if the same outcome model is used for both estimators (the g-computation "OLS" estimator and AIPW), AIPW is less precise but also less biased. G-computation is only consistent if the outcome model is correct (actually, there are some other technical cases in which it is consistent without the outcome model being correct, but they are unlikely, though perhaps no less likely than correctly specifying the outcome model). AIPW is consistent if either the outcome model or propensity score model is correct (again, there are technical cases that are more general than this statement). Typically, AIPW has less bias than g-computation, but that can depend on how poorly fit the propensity score model is. Still, I would tend to trust AIPW estimates over g-computation estimates.

Inference isn't just about the effect estimate, though; it is also about its variance. When the same outcome model is used for both, AIPW is less precise than g-computation (i.e., has a larger true variance). But the usual estimator of the AIPW variance, which uses influence functions, is only valid when both the outcome and treatment models are correct (i.e., AIPW estimates are doubly robust, but inference isn't). Similarly, the usual OLS variance for the g-computation estimator can be incorrect; it requires strict assumptions unlikely to be met in practice. Bootstrapping the standard errors for both methods is advisable.

So, the phenomenon you observe could be due to a number of factors. It could be due to the bias of the estimator, the error of the estimator (i.e., its inherent variability), and the accuracy of the variance estimate. There is no way to know which estimate is more accurate. One might require stricter assumptions to believe, but they both require simply having faith in the estimator. Unless you used a complicated estimator that makes a strong effort to attain asymptotic consistency of estimation and inference, or you use a method that allows you to assess the degree of structural bias remaining in the estimate, I wouldn't trust either estimate. You would need to provide more details about the specific models used to be able to assess how confident one could be in the estimate.

• Hi @Noah, thanks so much for the thorough reply. Do you have any references for some of your explanation? I would like to learn more. Aug 4, 2023 at 3:26