Assessing quality of propensity score matching via covariate balance using standardised difference

Question

I've been tasked with performing a propensity score matching and to measure the standardised difference for every covariate to assess the quality of fit. Details are given here and some SAS code here. For completeness, the standardised difference is defined for continuous covariates as: $$ \frac{(\bar{x}_1-\bar{x}_0)}{\sqrt{\frac{s^2_1+s^2_0}{2}}} $$ where $\bar{x}_1$ and $s^2_1$ are the mean and variance respectively of the covariate where the treatment is 1. If the covariate is dichotomous, the standardised difference is defined as: $$ \frac{|p_1 - p_0|}{\sqrt{\frac{p_1(1 - p_1)+p_0(1 - p_0)}{2}}} $$ where $p_1$ is the proportion of exhibiting cases where the treatment is 1.

If there is a small difference between the means of a covariate split by treatment, then that covariate could be considered balanced. However, I've been asked to use a cutoff of "10%", i.e. covariates are considered balanced if the standardised difference is less than 0.1. This seems a little odd to me since this will surely be biased; it will be dependent on the size of the mean and therefore is not bounded.

For example, take the normal distribution of mean $1 \times 10^9$, and $1 \times 10^{-18}$, and another with mean $0$ and variance $1 \times 10^{-18}$. The standardised difference will therefore be: $$ \frac{1 \times 10^9}{\sqrt{1 \times 10^{-18}}}= 1 \times 10^{18} $$ In order to compare the differences between this and another covariate (probably with a much smaller mean) to be compared, surely some other normalisation is required...I've not been instructed to standardise/centre my variables so am therefore worried about using this measure.

Do you have any criticism of this? Have I misunderstood this measure? If not, are there more common/appropriate measures to use to assess covariate balance?

Answer 1

If you are correct in your assumptions that the PS is additive and linear in the baseline variables and you have correctly fitted the PS (i.e., you didn't delete any "insignificant" variables) then this assumption does not need checking. An imbalance would imply an error in estimating a regression coefficient.

Making the same assumptions as listed above, a more meaningful examination of the data is to show that the entire distribution of continuous variables are identical. It is not enough to show that the means are equal.

I doubt that matching on PS is optimal. Can you be sure that matching did not discard any potentially matchable observations? Discarding data is not a good idea. Can you confirm that your matching algorithm is invariant to random dataset reordering? I prefer covariate adjustment for the logit propensity score, and expanding it into a regression spline so as to not assume linearity in the logit. Non-overlap regions should be checked for using the covariate approach. See http://www.citeulike.org/user/harrelfe/article/13340175 .