I have a small observational dataset (200 non-treatment, 100 treatment). It is obvious that the cohort of people receiving intervention were already sicker across the board by every metric. They also have worse outcomes. The tricky objective is to assess the role of the intervention.
Originally, I was going to be descriptive, and merely say that clearly the group receiving intervention is sicker (based off of comorbidities) and they happen to have worse outcomes despite intervention. Reading more, propensity score matching (PSM) seems to be the method of choice here.
I did my matching using the MatchIt package in R, (exact matching on 2 variables, nearest neighbor matching on 5 variables, logistic model, no caliper). Exact matching forced me to 90 observations in each group and was necessary because those two variables were most likely indicators of 'sickness' and therefore to be deemed in need of treatment.
Given that my objective is to adjust the two groups as best I can (but doesn't need to be perfect), is it okay if my balance is not great? Comparing slightly imbalanced groups seems better than completely imbalanced groups as in the unprocessed dataset. There are some patients who are just too sick and don't have a counterpart.
I chose no caliper because of the usual bias-variance trade-off; I'd exclude too many observations if I went with a small(er) caliper) which leads to unstable estimates. Given that there is a random component to MatchIt, is the correct approach to simulate it 1000 times and choose the most common matching?
If propensity score matching is not appropriate because of the small sample size (which is contributing to problems (1) and (2), what alternative is there beyond merely descriptive?
After matching, is using chi-square/Fisher's exact test/usual significance testing appropriate for assessing balance? I've seen some papers that suggest that only Average Absolute Difference, Q-Q plot, and small mean differences are okay for assessing the balance. This is best significance testing also depends on sample size. But as I said, so long as the balance is better than the original, that would be good in my opinion. But perhaps I am missing something.