Matching covariate selection- should we not match on binary features where overwhelming majority of subjects fall in one category?

I am trying to investigate if there is a relation between Occupational Therapy (OT) dosage for stroke patients and patient recovery. I have separated the patients into 2 groups by the amount of therapy they received, Low Dose vs High Dose, and am testing for inter-group differences. Because patients were not randomly assigned to the 2 groups and I am using data from a different study, we have to account for confounding factors separating the groups. We do so by using Genetic Matching and ensuring the Low Dose and High Dose groups are balanced in their observed covariates (features about patients). We tried to include as many covariates as we could as Matching covariates, but had to leave out some binary features because the subjects overwhelmingly fell in just one category.

For example: Only 5% of the patients had their preferred language as Spanish as opposed to English. If we included preferred language as a matching feature, it would be hard to find a similar (19:1) balance in both the Low Dose and the High Dose group, given that our Matching methods were also trying to simultaneously find balance across 20 other covariates.

This brings me to a fundamental question: For Binary features where the overwhelming majority of subjects fall in one category, how big should this discrepancy be before we start leaving features out as they lead to imbalanced groups after matching. I cannot arbitrarily come up with a number like 10%. I feel like there has to be some justification for selecting a threshold, but can't find literature on this anywhere.

The bias due to an imbalanced covariate is a function of the difference in means (i.e., proportions) between the two treatment groups. If the percentage of Spanish speakers in one group is around 90% in one group, there is no problem if the percentage is close in the other group. It doesn't matter whether the proportion in each group is close to 100% or close to 50%; all that matters is whether the proportions are similar in the two groups. It's not clear to me why you say

it would be hard to find a similar (19:1) balance in both the Low Dose and the High Dose group

It would seem very easy to find similar balance if the groups are initially balanced on this covariate. If being a Spanish speaker has little effect on whether the low or high dose was taken, then the groups will likely be very similar with regard to this variable, in which case it will be very easy to find balance on this variable (i.e., you might not even need to intentionally match on it to achieve balance). Genetic matching weights each variable based on whether prioritizing that variable yields balance overall. I would imagine that this variable will receive a small weight and not interfere with the genetic match at all.

That said, there are other matching algorithms you might try if genetic matching is not working for this specific problem. You could try propensity score matching, which, although less sophisticated, can often yield good performance. You could also try cardinality matching with the designmatch package, which uses optimization to find an optimally sized matched sample that satisfies given balance constraints.

• I guess I was thinking about it in a wrong way. One of the methods I am using to assess balance for binary variables is performing a KS test seeing and seeing how large the p-value is, but your logic should apply to that as well. Thank you for the extra suggestions. Propensity score matching was the first method we tried and was not leading to better balance (was actually increasing imbalance for a few variables). I haven't come across cardinality matching before but I'll look into it – stats_nerd Apr 21 at 7:59

The analysis started off on the wrong foot by thinking of matching. Matching will result in discarding some patients who are good matches, which scientifically is hard to defend and is arbitrary. This problem would be better addressed by all-comers and using covariate adjustment after verifying that there is no significant non-overlap region. (If there is a covariate that has an interval for which no patients received one of the treatment, you might restrict the overall analysis to the complementary interval).

The treatment received in your example is really time-dependent and may be possibly better analyzed with a time-dependent covariate analysis.

Your study is of the type where you expect a significant survival bias to exist, so it may be reasonable to restrict the analysis to patients who survival 1 or 2 months.

I suggest working with an epidemiologist who is versed in causal inference.

• Are there cases where matching should be used or covariate adjustment? – usεr11852 Apr 21 at 22:29
• It is true that matching (with replacement) is adding bias and not using all the subjects. However, I am not sure I followed what you meant by "it may be reasonable to restrict the analysis to patients who survival 1 or 2 months.". Could you please elaborate on that? – stats_nerd Apr 22 at 10:58
• The time restriction could be said to constitute a landmark analysis where you restart the clock so that going forward in an equal-opportunity fashion. In other words, if you lose patients for any reason before they qualify to be called "under treatment X" there is a survival bias that needs to be accounted for. Regarding matching, match only when covariate adjustment is impossible, e.g., when studying identical twins. See Section 10.1 of BBR for more about problems with matching. – Frank Harrell Apr 22 at 12:39