I've been tasked with performing a propensity score matching and to measure the standardised difference for every covariate to assess the quality of fit. Details are given here and some SAS code here. For completeness, the standardised difference is defined for continuous covariates as: $$ \frac{(\bar{x}_1-\bar{x}_0)}{\sqrt{\frac{s^2_1+s^2_0}{2}}} $$ where $\bar{x}_1$ and $s^2_1$ are the mean and variance respectively of the covariate where the treatment is 1. If the covariate is dichotomous, the standardised difference is defined as: $$ \frac{|p_1 - p_0|}{\sqrt{\frac{p_1(1 - p_1)+p_0(1 - p_0)}{2}}} $$ where $p_1$ is the proportion of exhibiting cases where the treatment is 1.

If there is a small difference between the means of a covariate split by treatment, then that covariate could be considered balanced. However, I've been asked to use a cutoff of "10%", i.e. covariates are considered balanced if the standardised difference is less than 0.1. This seems a little odd to me since this will surely be biased; it will be dependent on the size of the mean and therefore is not bounded.

For example, take the normal distribution of mean $1 \times 10^9$, and $1 \times 10^{-18}$, and another with mean $0$ and variance $1 \times 10^{-18}$. The standardised difference will therefore be: $$ \frac{1 \times 10^9}{\sqrt{1 \times 10^{-18}}}= 1 \times 10^{18} $$ In order to compare the differences between this and another covariate (probably with a much smaller mean) to be compared, surely some other normalisation is required...I've not been instructed to standardise/centre my variables so am therefore worried about using this measure.

Do you have any criticism of this? Have I misunderstood this measure? If not, are there more common/appropriate measures to use to assess covariate balance?


If you are correct in your assumptions that the PS is additive and linear in the baseline variables and you have correctly fitted the PS (i.e., you didn't delete any "insignificant" variables) then this assumption does not need checking. An imbalance would imply an error in estimating a regression coefficient.

Making the same assumptions as listed above, a more meaningful examination of the data is to show that the entire distribution of continuous variables are identical. It is not enough to show that the means are equal.

I doubt that matching on PS is optimal. Can you be sure that matching did not discard any potentially matchable observations? Discarding data is not a good idea. Can you confirm that your matching algorithm is invariant to random dataset reordering? I prefer covariate adjustment for the logit propensity score, and expanding it into a regression spline so as to not assume linearity in the logit. Non-overlap regions should be checked for using the covariate approach. See http://www.citeulike.org/user/harrelfe/article/13340175 .

  • $\begingroup$ Thanks for your response Frank. Unfortunately I have no choice about the method being used here and just wanting to gauge opinion about the standardised difference statistic. Assuming that the PS was correctly modelled, would the SD be an appropriate way to compare the balance of covariates i.e. if covariate a has an s.d. = 0.5, and covariate b has an s.d. = 0.1, covariate b is 'more balanced' than a $\endgroup$ – James Owers Sep 1 '14 at 20:06
  • $\begingroup$ What is the reason you do not have a choice? Have you verified that no matchable observations were removed, thus making the approach less scientific? Or that there are no arbitrary dataset order effects? And as I said earlier you need to look at the entire distribution using the empirical cumulative distribution function (two ECDFs overlayed on one plot for each continuous variable). $\endgroup$ – Frank Harrell Sep 1 '14 at 20:09
  • $\begingroup$ Thanks again for your input. The reason I have no choice is that I'm following a brief assigned by my client! Have you any comment about the statistic? $\endgroup$ – James Owers Sep 1 '14 at 20:12
  • $\begingroup$ Note that the approach could be attacked in court. Regarding the statistic, the standard deviation is only applicable when the variable has a symmetric distribution, and standardization of variables is an arbitrary approach I would not recommend. I would look for imbalance in real units plus use the ECDF. $\endgroup$ – Frank Harrell Sep 1 '14 at 20:19
  • $\begingroup$ Cheers Frank, appreciate your comments. Will look into the article you cited and also investigate CBPS which seems an altogether better way to go about things $\endgroup$ – James Owers Sep 1 '14 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.