2
$\begingroup$

I have a data set from a cohort comparing two treatmens which I want to balance via propensity score matching. I read some literature and decided to use a variable K:1 matching because this seems to have some advantages compared to a fixed 1:1 matching, particularly less reduction in the sample size. To ensure balance after matching I want to use a caliper. My first question is about the caliper: In matchIt the default setting is std.caliper = TRUE, so for my understanding this means, that if I use 0.2 there, I'm using actually 0.2 * SD of the PS. However, I found examples, where 0.2 * sd(logitPS) is used (and this option is still TRUE), which confuses me. Further, 0.2 * SD of the PS is not the same as 0.2*sd(logitPS), so I'm not entirely sure, which I should use. An additional question on that would be whether it makes sense to use the same caliper for all (9) variables, although some of them are binary and some are continuous.

The next thing I'm not really sure is the performance of the logistic model estimating the propensity score. Regarding its diagnostics it seems to be a pretty poor model, e.g. Pseudo-R² (Cragg-Uhler) = 0.11 and a lot of the varibles are not significant (I know, this schould be evaluated with caution and I won't perform any varible reduction here). However, I'm wondering whether this is okay (given that the matching itself works properly and produces balanced treatment groups) or whether I should try to put even more effort in a better fit.

Finally, I use the package cobalt to check whether my sets are balanced and to produce corresponding plots. The default is that for binary variables raw differences are used and for continuous ones standardized differences. Which should I present in the tables and in the (love) plots? This can make a difference when having values very close to the margin of 0.1.

Hopefully here are some experts on that topic. Thanks a lot in advance!!

$\endgroup$

1 Answer 1

1
$\begingroup$

You should make sure your sample isn't balanced before using a caliper. If a caliper isn't necessary, using a caliper makes balance worse and unnecessarily decreases your sample size and prevents generalizability to a meaningful population. Using a caliper should not be a default (which is why it isn't in MatchIt).

If you decide to use a caliper, you can place one on the propensity score, on the covariates directly, or both. If you use a caliper on the propensity score, it is recommended instead to match on the logit of the propensity score. In MatchIt, this can be requested by setting link = "linear.logit" in the call to matchit(). I would recommend doing so to be consistent with the recommendations in the literature. Of course, you should choose the option that yields the best balance, effective sample size, and representativeness; what you find in the literature may not match the scenario you observed in your own dataset.

Similarly, the size of the caliper on the propensity score (logit) or on the covariates cannot be determined by rules in the literature and instead must be evaluated based on the features of your dataset. If some covariates are not being balanced by the matching, it may be effective to place a caliper on those covariates directly. This can help overcome a poorly fitting propensity score model. You cannot place a caliper on a categorical covariate; instead, you can perform exact matching on that covariate by supplying it to the exact argument. You should adjust the size of the caliper to fit the needs of your dataset, which are determined by the remaining imbalance and the standardized pair differences in the MatchIt summary() output. There is no one single way to do matching best; figuring out the optimal specification is an art, though there are ways to reduce the human involvement in it, e.g., by using a more sophisticated matching algorithm that seeks balance directly.

The choice of whether to use and display raw differences in proportion or standardized mean differences is up to you and your readers. I lay out my justification for using raw differences in proportion in the cobalt documentation. If any balance statistic is borderline, then improve balance on that covariate. Don't worry too much about the specifics of how a statistic is calculated. To be conservative, use standardized mean differences for all covariates. The problem is that it is possible to have a raw difference in proportion that is very small and substantively meaningless but for it to have a large standardized mean difference because the standardization factor can arbitrarily magnify the observed statistic. To avoid that, use raw differences in proportion. Whatever you do, make sure you report it correctly in your manuscript.

$\endgroup$
9
  • $\begingroup$ Dear Noah, thanks a lot for your answer. When using no caliper, 4 of my 9 variables are not balanced after matching- this is why I want to use one. I understand that the choice of the caliper is depending on my data. However, I still don't understand the syntax with regard to "std.caliper" - why is this option still true wenn using a caliper of 0.2*sd(logitPS)? I decided to put my caliper on the propensity score. Using this with 1:5 variable matching ratio I don't loose any observation at all, this appears strange to me and brought up the question of the correctly specified caliper... $\endgroup$
    – Kathrin
    Commented Jul 13, 2023 at 16:34
  • $\begingroup$ A caliper doesn't necessarily drop observations; it just prevents some units from being paired together. std.caliper is TRUE because you want .2 standard deviations of the logit of the propensity score. If you set std.caliper = FALSE, then you would have a caliper of .2 logits of the propensity score. link and std.caliper have nothing to do with each other; one transforms the propensity score (into a logit) and one changes the scale of the caliper (to be on the standard deviation scale instead of the raw scale). $\endgroup$
    – Noah
    Commented Jul 13, 2023 at 18:05
  • $\begingroup$ Hi Noah, I think there's a misunderstanding, I'm referring to this code which I found in a very good manual: matchit(ps.formula, data = analytic, distance = 'logit', method = "nearest", replace=FALSE, caliper = .2*sd(logitPS), ratio = 1), where std.caliper = TRUE as this is the default. I don't get this, from my understanding this is now the standard deviation of the standard deviation? I searched a lot but I didn't find an explanation for that. I thought, std.caliper = TRUE would AUTOMATICALLY take the standard deviation of the logit of the PS, so I would choose a caliper of 0.2 then. $\endgroup$
    – Kathrin
    Commented Jul 13, 2023 at 18:52
  • $\begingroup$ I totally agree to your remaining comments and I'm very thankful for that. I see clearer now and use link = "linear.logit" for my caliper. However, I'm still unsure how to choose this caliper due to the problem mentioned above. Using no caliper doesn't work due to remaining inbalances. Also, as I mentioned, my logistic model is not very good, which could also be an explanation. I could slightly improve it by adding an interaction term, but still, without setting a caliper I don't get a balanced data set. $\endgroup$
    – Kathrin
    Commented Jul 13, 2023 at 18:56
  • $\begingroup$ Finally, I just think it's an error in the documentation which confused me over two days. Did I everything correctly when using: match.obj <- matchit(ps.formula2, data = data, distance = 'glm', method = "nearest", link = "linear.logit", replace=FALSE, caliper = .2, std.caliper = TRUE, ratio = 5) Because this results in a very nice balanced data set and a very very small reduction. (I tried values from 0.1 to 0.3 and this was the "perfect" value regarding balance AND size). $\endgroup$
    – Kathrin
    Commented Jul 13, 2023 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.