# When propensity matching doesn't work

A clinical group presented a dataset based on a convenience sample of about $n=600$ patients in 3-groups, with roughly $n=200$ in each group. Like a lot of groups, the request was "we want to use propensity matching to make the data more like a trial." Which really means to reduce confounder influences across the groups. To attack the problem, the following steps were taken:

1. First, ANOVA (KW) was used to determine which potential baseline covariates were significantly different across the 3 treatment groups. Several were identified. Welch ANOVA was used if Bartlett's test was significant.
2. Next, polytomous regression was used with the treatment group variable as the dependent and significant covariates as independent variables to determine the predictability of treatment group membership for subjects in each of the three treatment groups. It worked out that group membership prediction for only a small number of subjects in treatment group 1 was accurate, likely meaning that the significance during ANOVA was due to either differences between treatment groups 2 and 3, and group 1 was just noise.

Given this dilemma, the result indicate that a low chance of finding covariate values (i.e., group 1 logit values among group 1 subjects) in treatment group 1 subjects which are similar to group 2 and group 3-based logit values in group 2 and 3 subjects, respectively.

Theoretically, if gender is a covariate, and the data were from a randomized control trial (subjects randomized to treatment groups) with an infinitely large sample size, you would be able to find an adequate number of males in treatment groups 1,2,3 and females in groups 1,2,3 whose logit values could be matched.

Here's the bad news. The treatment group 1 subjects are low risk for which standard of care was used, treatment group 2 was moderate therapy, and treatment group 3 was aggressive treatment. Therefore, the treatment group variable is dependent on disease severity, or low, moderate, and high risk. The problem which is occurring is that the pre-treatment assignment to treat with standard of care (group 1) for low risk-severity subjects drives their baseline covariate values to take on characteristics different group 2 and 3 covariate patterns.

Essentially, the problem is that propensity matching is meant for covariates whose patterns can be matched in different treatment groups (i.e., randomized trials, for which treatment is not related to disease severity). In addition, most groups use propensity matching with trial data and then split the dataset from a different angle using a confounder such as low risk/high-risk based on groups or a median split of a continuous covariate.

In summary, I believe propensity matching is not preferred for convenience-sample data and a grouping variable which is the treatment choice dependent on disease severity (risk), as it results in unbalanced covariate patterns which are difficult to match when using the logits for matching.

QUESTION: In light of the above described attempt to evaluate the ability to perform propensity matching for such a convenience sample, is there an alternative approach to propensity matching?

• I tend to be pretty skeptical of propensity score matching myself. I think it's mostly people fooling themselves by finding a method that promises an easy statistical trick that lets them say what they want to. (That said, there's no question it works on paper.) In your case, can you do a reasonable job of measuring disease severity, or estimating it as a latent variable? Regression discontinuity sounds like it might be promising here. – gung Apr 15 '18 at 16:38
• I think that a core question is what should propensity matching do when working. It is just that the marginal distribution of the underlying covariates looks the same? Is it correct inference based on simple means? PSM is not the "end game". – usεr11852 May 6 '18 at 20:57