# Evaluating propensity score matches- what to do when ratio of variances or standardized means of difference go to infinity?

I am working on a project where I am comparing the effects of a particular treatment on patients with other patients who didn't receive the treatment. As I am trying to replicate a randomized experiment using data from a observational experiment, I am using matching (I'm trying propensity score, euclidean and mahalanobis distance matching separately to see what produces the best matches) to find a subset of the control group that is best-matched to my treatment group. This essentially means finding control patients that are most similar to the treated patients, according to the available data (demographics and testing data).

For evaluating how well the groups are matched, it is common to look at the ratio of the variances for each co-variate between the treated and the matched control group, the ideal range of ratios being close to 1. It is also common to look at the standardized difference of means for each co-variate between the treated and matched control group, the ideal values considered to be less than 0.25 and as close to 0 as possible.

The problem I am coming across is that some of these covariates end up having a variance of 0 either in the treated set or the matched-control set. Which means all the patients in either the treated or control group are identical in terms of that co-variate. This means the standardized difference of means becomes infinity for a covariate when the standard deviation for that co-variate is 0 in the treated group. It also means that the ratio of variances can lead to a value of 0 or infinity.

What is the proper way to deal with such cases?

## 1 Answer

The effect of imbalance on the bias of the effect estimator has to do with the difference in means of each pretreatment covariate relevant to the outcome and the strength fo the relationship between the covariate and the outcome. We use standardized mean differences because it allows us to apply the same metric across all covariates and we can easily define standards for imbalance that don't require subject-matter knowledge for each variable. But if you knew exactly how much imbalance in the raw units of the variable would cause significant bias in the estimate, there would be no need to standardize the mean difference. For example, if the outcome depends heavily on age, an imbalance of 5 years between the treated and control might be too large. Dividing this mean difference by the standard deviation in the sample doesn't get you any closer to eliminating and predicting bias in your effect estimate; it's the mean difference that matters.

All this is to say that for the variable you can't standardize, don't standardize them. Look at the raw mean difference and try to make a conclusion about whether the raw mean difference indicates significant imbalance. You, of course, will have to back up your decision with substantive reasoning, but that shouldn't be too much of a challenge. Instead of examining variance ratio, you could examine the mean difference of the square of the (centered) variable, which gets you at the same diagnostic without having to take a ratio.

Another possibility is to estimate the standard deviation using a different population for which there is nonzero variance. If that population is the same population from which your sample was drawn, its standard deviation would be a good substitute for the sample standard deviation (which is zero).

One final thought is that the denominator of the standardized mean difference should be the value in the unmatched sample. Otherwise, changes in the mean difference are conflated with changes in variance.