Alternative analysis to propensity score matching for small sample sizes?

Question

I have a small observational dataset (200 non-treatment, 100 treatment). It is obvious that the cohort of people receiving intervention were already sicker across the board by every metric. They also have worse outcomes. The tricky objective is to assess the role of the intervention.

Originally, I was going to be descriptive, and merely say that clearly the group receiving intervention is sicker (based off of comorbidities) and they happen to have worse outcomes despite intervention. Reading more, propensity score matching (PSM) seems to be the method of choice here.

I did my matching using the MatchIt package in R, (exact matching on 2 variables, nearest neighbor matching on 5 variables, logistic model, no caliper). Exact matching forced me to 90 observations in each group and was necessary because those two variables were most likely indicators of 'sickness' and therefore to be deemed in need of treatment.

1. Given that my objective is to adjust the two groups as best I can (but doesn't need to be perfect), is it okay if my balance is not great? Comparing slightly imbalanced groups seems better than completely imbalanced groups as in the unprocessed dataset. There are some patients who are just too sick and don't have a counterpart.

2. I chose no caliper because of the usual bias-variance trade-off; I'd exclude too many observations if I went with a small(er) caliper) which leads to unstable estimates. Given that there is a random component to MatchIt, is the correct approach to simulate it 1000 times and choose the most common matching?

3. If propensity score matching is not appropriate because of the small sample size (which is contributing to problems (1) and (2), what alternative is there beyond merely descriptive?

4. After matching, is using chi-square/Fisher's exact test/usual significance testing appropriate for assessing balance? I've seen some papers that suggest that only Average Absolute Difference, Q-Q plot, and small mean differences are okay for assessing the balance. This is best significance testing also depends on sample size. But as I said, so long as the balance is better than the original, that would be good in my opinion. But perhaps I am missing something.

Answer 1

1) If your goal is to make a causal inference, balance is paramount. Although you may have improved balance, if it is not good then your causal inference may still be invalid (/your estimate will still be biased). If you have untreated units that fall outside the range of your treated units, your causal inferences will not be valid for them unless you can justify extrapolation. You may want to delete these cases and limit your inferences to the region of overlap. (Overlap can be conceptualized as the overlap between the covariate distributions, e.g., the convex hull, or common support on the estimated propensity scores.)

2) You could randomly simulate, but I think a better approach would be to find the matched groups that yield the best balance and move forward with that single sample.

3) I'm not sure what your outcome is, but an "alternative to matching" is regression (to many, matching is an alternative to regression). If you are willing to make parametric assumptions and use regression of some kind (e.g., logistic, linear, count), you can use regression instead of or in addition to matching. With only 7 covariates that you want to control for, it shouldn't be too hard to run a regression and account for potentially relevant interactions and curviliearities.

4) Hypothesis tests are NOT appropriate for assessing balance. Balance is a sample property, so there is no sense in which a p-value will be more helpful than an effect size measure. Also, typically, balance is achieved by comparing your balance statistics to a chosen threshold, not by looking at reduction in imbalance from your original sample. The fact that you've achieved balance slightly better than you started with doesn't mean you can move forward; you need to arrive at balance that permits an unbiased estimate of the treatment effect.

I recommend you try various methods of conditioning on the propensity scores. You've chosen matching, but there doesn't seem to be reason not to try weighting or full matching (really a form of weighting). Weighting using CBPS or by entropy balancing can be very effective. If you want to compare balance across multiple methods of conditioning, you can use the cobalt package which interfaces with some of the other packages and offers some additional tools for balance assessment. I also recommend you combine propensity score conditioning with regression on the treatment and your covariates. That technique is preferred in the literature and can reduce the remaining imbalance in your adjusted sample.