# In propensity score analysis, what are options to deal with very small or large propensities?

$\newcommand{\P}{\mathbb{P}}$I am concerned with observational data in which the treatment assignment can be explained exceedingly well. For example, a logistic regression of

$$\P(A =1 |X) = (1+ \exp(-(X\beta)))^{-1}$$

wehre $A$ treatment assignment and $X$ covariates has very good fit with very high test $AUC >.80$ or even $>.90$. This is good news for the accuracy of the propensity model, but it leads to propensity score estimates $$\hat{\pi} = (1+ \exp(-(X \hat{\beta})))^{-1}$$ close to $0$ or $1$. These in turn lead to large inverse probability weights $\hat{\pi}^{-1}$ and $(1-\hat{\pi})^{-1}$ used in estimators such as the inverse probability weighted estimator of expectation of outcome $Y_1$ (observation under treatment):

$$n^{-1} \sum_i \hat{\pi_i}^{-1} A_i Y_{1i}.$$

This, I suspect, turns the estimates' variances very large.

It seems like a vicious circle that very discriminative propensity score models lead to extreme weights.

My question: what are available option to make this analysis more robust? Are there alternatives to fit the propensity score model or how to deal with large weights after the model has been fit?

• You might want to take a look the covariates carefully. You should include all of the variables that affect both (not either, but both) participation and outcomes. Including ones affected by the treatment, either ex post or ex ante in anticipation of treatment, is bad. In particular, Including instruments – variables that affect participation and not outcomes – is also a particularly bad idea. They will not help with selection bias and may worsen the support problem drastically. For example, if some people are encouraged to take up treatment, you don't want to condition on that. Nov 22, 2016 at 18:01
• @DimitriyV.Masterov Thanks; your last points seems interesting/relevant to my situation. So are you saying it is best not to find the best treatment assignment model (but rather the one including the predictors of outcome and assignment)? I thought the more precisely we can predict assignment, the better. Nov 22, 2016 at 19:18
• I think that is a common misconception. For example, see Battacharya and Vogt (2012) paper in International Journal of Statistics and Economics on the instruments point. Nov 22, 2016 at 19:29
• @DimitriyV.Masterov while your answer may solve the problem of small propensities in some situations, it may still be the case that the set of $X$ relating to both $Y$ and $A$ is very discriminative on $A$. I am still interested in options to deal with this problem. Nov 24, 2016 at 15:28