Theoretical question about post-matching analysis of propensity score matching

Question

I have been developing a propensity score matching model, using logistic regression to model propensity, and I am wondering about recomputing propensity scores in order to validate a model. Here is my hypothesis, please tell me where I am wrong: If propensity score matching "works" and we have a control set that is balanced on the treatment set, if we recompute the propensity scores and plot them both, they should be normally distributed with mean 0.5.

I took some data I was using, and constructed a control set in the following way: found the mean and variance of each covariate in the trial set and then made a control set that randomly samples from the normal distribution but with treatment = 0. When I did that, sure enough, the propensity scores were distributed around mean 0.5.

This makes sense to me because if you have adjusted for covariate balance, then it is a coin flip whether they received treatment or not, right? And using logistic regression, there is basically no signal as to whether a person is treated or not, so p ~ 0.5.

If so, why wouldn't a diagnostic of whether propensity score matching worked be to recompute the propensity scores for the constructed control group and see if they cluster around mu = 0.5?

Answer 1

This is a good thought, but the problem arises when you have important covariate imbalances that are not displayed by this method. Consider two variables on the same scale that have different effects on the outcome and the same effect on the treatment probability. If they are imbalanced with the same magnitude in the matched set, the marginal propensity score mean is .5, but you would expect to see bias in the effect estimate.

Consider the following example:

> x1 = rnorm(n)
> x2 = rnorm(n)
> z = as.numeric(x1 + x2 > rnorm(n))
> tapply(x1, z, mean)
         0          1 
-0.4991431  0.4709013 
> tapply(x2, z, mean)
         0          1 
-0.4262566  0.4316203 
> ps = glm(z ~ x1 + x2, family = binomial(link = 'probit'))$fitted
> mean(ps)
[1] 0.4990706

The average propensity score (ps) is .5 in this data set, and yet there are large imbalances in the covariates. If there was not perfect bias cancellation (i.e., if the variables affected the outcome differently), you would still see bias in the effect estimate. It may be that a mean propensity score of .5 is necessary for balance, but it's definitely not sufficient. Either way, it always makes sense to assess balance on the covariates directly.