Using a caliper means that some units will not be matched and therefore receive a weight of 0. When matching for the ATT, when a treated units is dropped due to the caliper, we know that a unit that would have had a weight of 1 instead receives a weight of 0. That can shift the distribution of the covariates away from that of the treated group if enough treated units are dropped, which is why using a caliper when targeting the ATT can change the estimand.
When full matching for the ATE, we don't know what weight a unit would have had had there been no caliper. The weight for a unit is equal to the inverse of the proportion of units of the same group in that unit's subclass. For example, in a subclass with 3 treated units and 7 control units, each treated unit would receive a weight of 1/.3 and each control unit would receive a weight of 1/.7. (Note that all weights are greater than 1, but what's important is the relative weight compared to others in the same group.) A treated unit in a group with a high proportion of treated units receives a smaller weight. There isn't much difference between receiving a small weight and receiving a weight of 0 (i.e., due to being removed from a caliper). So if the caliper only drops units that would have had a small weight otherwise, the caliper won't have a large effect on balance, precision, or the estimand. If the caliper drops units that would have had a large weight otherwise, then it will have a large effect on these qualities and may change the estimand.
The way to know whether the estimand has meaningfully changed is to look at the distribution of covariates before and after matching. Normally we only assess balance by comparing the covariates between the treatment groups, but you can also assess generalizability by comparing the distribution of covariates in each group to the distribution of covariates in the full sample. This isn't straightforward in MatchIt
, but in cobalt
, you can just run bal.tab(., pairwise = FALSE, which.treat = .all)
to see how different each treatment group is from the full sample after matching. If you notice a large shift from the covariate distribution in the full sample when matching with a caliper, you are no longer targeting the ATE. If the distribution after matching is similar to the distribution in the full sample, then it is likely you are still targeting the ATE.