Propensity scores: Variable K:1 matching with caliper- some technical questions (MatchIt, cobalt)

Question

I have a data set from a cohort comparing two treatmens which I want to balance via propensity score matching. I read some literature and decided to use a variable K:1 matching because this seems to have some advantages compared to a fixed 1:1 matching, particularly less reduction in the sample size. To ensure balance after matching I want to use a caliper. My first question is about the caliper: In matchIt the default setting is std.caliper = TRUE, so for my understanding this means, that if I use 0.2 there, I'm using actually 0.2 * SD of the PS. However, I found examples, where 0.2 * sd(logitPS) is used (and this option is still TRUE), which confuses me. Further, 0.2 * SD of the PS is not the same as 0.2*sd(logitPS), so I'm not entirely sure, which I should use. An additional question on that would be whether it makes sense to use the same caliper for all (9) variables, although some of them are binary and some are continuous.

The next thing I'm not really sure is the performance of the logistic model estimating the propensity score. Regarding its diagnostics it seems to be a pretty poor model, e.g. Pseudo-R² (Cragg-Uhler) = 0.11 and a lot of the varibles are not significant (I know, this schould be evaluated with caution and I won't perform any varible reduction here). However, I'm wondering whether this is okay (given that the matching itself works properly and produces balanced treatment groups) or whether I should try to put even more effort in a better fit.

Finally, I use the package cobalt to check whether my sets are balanced and to produce corresponding plots. The default is that for binary variables raw differences are used and for continuous ones standardized differences. Which should I present in the tables and in the (love) plots? This can make a difference when having values very close to the margin of 0.1.

Hopefully here are some experts on that topic. Thanks a lot in advance!!

Answer 1

You should make sure your sample isn't balanced before using a caliper. If a caliper isn't necessary, using a caliper makes balance worse and unnecessarily decreases your sample size and prevents generalizability to a meaningful population. Using a caliper should not be a default (which is why it isn't in MatchIt).

If you decide to use a caliper, you can place one on the propensity score, on the covariates directly, or both. If you use a caliper on the propensity score, it is recommended instead to match on the logit of the propensity score. In MatchIt, this can be requested by setting link = "linear.logit" in the call to matchit(). I would recommend doing so to be consistent with the recommendations in the literature. Of course, you should choose the option that yields the best balance, effective sample size, and representativeness; what you find in the literature may not match the scenario you observed in your own dataset.

Similarly, the size of the caliper on the propensity score (logit) or on the covariates cannot be determined by rules in the literature and instead must be evaluated based on the features of your dataset. If some covariates are not being balanced by the matching, it may be effective to place a caliper on those covariates directly. This can help overcome a poorly fitting propensity score model. You cannot place a caliper on a categorical covariate; instead, you can perform exact matching on that covariate by supplying it to the exact argument. You should adjust the size of the caliper to fit the needs of your dataset, which are determined by the remaining imbalance and the standardized pair differences in the MatchIt summary() output. There is no one single way to do matching best; figuring out the optimal specification is an art, though there are ways to reduce the human involvement in it, e.g., by using a more sophisticated matching algorithm that seeks balance directly.

The choice of whether to use and display raw differences in proportion or standardized mean differences is up to you and your readers. I lay out my justification for using raw differences in proportion in the cobalt documentation. If any balance statistic is borderline, then improve balance on that covariate. Don't worry too much about the specifics of how a statistic is calculated. To be conservative, use standardized mean differences for all covariates. The problem is that it is possible to have a raw difference in proportion that is very small and substantively meaningless but for it to have a large standardized mean difference because the standardization factor can arbitrarily magnify the observed statistic. To avoid that, use raw differences in proportion. Whatever you do, make sure you report it correctly in your manuscript.