On https://cran.r-project.org/web/packages/cem/vignettes/cem.pdf, it mentions that "Using the output from cem, we can estimate SATT via the att function. The simplest approach requires a weighted difference in means (unless k2k was used, in which case no weights are required). For convenience, we compute this as a regression of the outcome variable on a constant and the treatment variable, where the SATT estimate is the coefficient on the treated variable. The function att allows for R’s standard formula interface and, by default, uses a linear model to estimate the att using the weights produced by cem. In other situations, with some coarsening, some imbalance remains in the matched data... Thus, a reasonable approach in this common situation is to attempt to adjust for the remaining imbalance via a statistical model...To apply a statistical model to control for the remaining imbalance, we use the formula interface in att." Here, the SATT is sample average treatment effect on the treated and the k2k means one to one matched data rather than one to many. It is clear that the coefficient of lm with one predictor (the treated variable) is basically a difference in means. However, when there are other predictors, in the cem vignette, they go on to run a linear model using R's formula interface and use the coefficient on the treated variable only. In fact, only the coefficient on the treated variable is presented in the att function output.

My question is if I am manually doing this through the lm function, when doing analysis on matched data, using either propensity scores or coarsened exact matching, what part of the statistical machinery for the linear model run on the matched data should still be checked (i.e., diagnostics, etc.)? Because in the vignette, they don't present coefficients of other variables, does it mean it is OK for there to be, for example, many variables that were non-significant still present? Should we be checking things like multicollinearity via the variance inflation factor (vif)? They also don't seem to do any residual analysis. Is it because the variables are simply there to remove some remaining imbalance? My primary objective is the weighted difference in means (treated = 1 vs. treated = 0), i.e., the coefficient of the treated, not the coefficients of the other variables for the linear model. However, residual analysis would show if the p-values obtained on the assumption of normality of the residuals makes sense or not. What if residual analysis showed that the residual qq plot looked way off? Should I bootstrap (or use something like MASS::rlm) on the matched data or again, simply concern myself with the coefficient of the treated variable?


One of the authors of the coarsened exact matching (CEM) package for R responded to my question in relation to CEM. I will post his remarks here (with permission from him) and I think they answer my question quite satisfactorily:

"I'd separate the issue you raise in two: matching and lm regression.

As far as matching is concerned, several problems that are typical of regression analysis are not relevant. For example, if two collinear variables are included among matching variables, this does not (per se) yield any bias in result, because the two covariate induce the same match, given the assigned coarsening. On the other side, all the imbalance you have tolerated remains and MAY GENERATE bias in your estimates.

On the contrary, in the regression step all the concerns that are typical of regression analysis can be considered, including multicollinearity and the QQ plot residual analysis. These aspects, in fact, have an impact on all the estimated coefficient, including the coefficient connected to the treatment variable."

Regarding the "MAY GENERATE" part, he clarified saying

"when you choose a coarsening, you are assuming that:

a. the residual imbalance between treated and untreated units is not relevant to the att estimation, i.e. the difference between treated and untreated units does not generate effects on the outcome variable. In this case your analysis does not need any further step;


b. the residual imbalance generates bias and you can try to deal it with a statistical model (e.g. lm)".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.