Positivity, Causal Inference and Machine Learning methods

Question

I ran into a problem where the predicted probabilities of being treatment/control from a machine learning model (the XGB model) matches the actual outcome almost exactly and have area under the curve (AUC) of 0.9999. On the other hand, the logistic regression achieved an AUC of 0.5. However, I cannot use the greedy matching algorithm to find any matches for the treatment if I use the XGB propensity scores. One of the earlier posts here mentioned this problem: How to use propensity scores that exhibit separation. So my question is: in order for the matching algorithm to work, do we have to give up the better machine learning prediction model for an inferior logistic regression? This seems a little counterintuitive.

Answer 1

You are judging your propensity score models on the wrong criteria. The only criteria you should use to assess how good your propensity scores are are how well they achieve balance and the remaining effective sample size when conditioned on (i.e., using matching or weighting). I explain this in my answer here. You should not use AUC to assess propensity score models; the fact that XGB has a higher AUC than LR doesn't mean anything. It's simple to overfit a propensity score model so that you get perfect prediction of the treatment; this doesn't actually mean anything about positivity and doesn't mean the overfit model is preferable.