1
$\begingroup$

I've completed propensity score matching of a treatment and control group across a number of covariates. On two categorical covariates we require an exact match, for example, Gender and Eye Color, and we're trying to determine effect of a treatment on continuous feature like salary (silly example).

However, there is one group of, say females with brown eyes, that are under represented in the unmatched data. Because of the exact matching, this causes males with brown eyes to be under represented in the resulting matched data (regardless of their significant representation in the unmatched data). I'm trying to determine how this could be impacting our salary prediction and quantifying the potential error due to bias in the matched sample.

Put another way, what is the potential error in the matched data due to differences in covariate distribution between matched and unmatched samples?

I've gone down the path of assessing confounding variables and effect modification, but I'm not sure 1) if I can compare those effects across populations and 2) how I should present the information [1]. I might be over complicating, but it would seem that I need to adjust for the confounding features and than adjust the effect modification by the distribution in the matched data? I've also tried estimating treatment effects after matching [2], bootstrapping, and rbounds [3] but none seem to be addressing what I need-- I suspect that I'm just phrasing the question wrong or missing the obvious?

[1] https://sphweb.bumc.bu.edu/otlt/MPH-Modules/PH717-QuantCore/PH717_MultipleVariableRegression/PH717_MultipleVariableRegression5.html [2] https://cran.r-project.org/web/packages/MatchIt/vignettes/estimating-effects.html#after-pair-matching-without-replacement [3] https://cran.r-project.org/web/packages/rbounds/rbounds.pdf [4] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2943670/

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

When you do pair matching, i.e., where you find one (or more) control units for each treated unit, you are estimating the average treatment effect in the treated (ATT). The ATT is what it sounds like; it's the average effect of the treatment just for those who received the treatment. If the distribution of covariates among the treated units differs from that in the overall sample (and it will if there is covariate imbalance, which is why you would use matching in the first place), then the effect will not generalize to the whole sample (or the population from which the sample was drawn). The average treatment effect in the population (ATE) is the average effect of the treatment for all units in the population. When the effect of treatment depends on confounding covariates, the ATT and ATE will not coincide.

The implication is that pair matching cannot be used to estimate the ATE. It sounds like you want to estimate the ATE since you are worried that your matched sample is not representative of the unmatched sample, the latter of which is (ideally) representative of your population of interest. If you want to estimate the ATE, you can just use one of the many methods available that do estimate the ATE. Pair matching (as implemented in MatchIt) is not one of them. Among matching methods, you can use full matching, propensity score subclassification, or template cardinality matching. You can also use weighting methods like inverse probability weighting that target the ATE. You can also use matching imputation as implemented in the Matching package and teffects in Stata.

It's important to assess whether your matched or weighted sample resembles the population you want to generalize to, so it's good of you to have noticed that the covariate distribution in your matched sample differs from that in the original sample. The similarity between an analysis sample and a target population is called "target balance". Unless one's goal is to estimate the ATT or one doesn't care to which population one's effect generalizes, one should always check target balance. When treated units are discarded as part of the matching (i.e., because they couldn't find matches due to an exact matching restriction or caliper), the resulting matched sample may not even resemble the treated units, in which case neither the ATT nor ATE is being estimated. I discuss that here.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I'm following correctly then matched pair can only est. treatment effect within the matched population but is not generalizable to the unmatched population (ATE). As it stands, I believe the matched population is the maximum number of matches possible for the dataset (which was our original focus). If we work to optimize representation within the matched population we'll really just be dropping matches till the proportions of the matched population reflect the unmatched. This would seemingly challenge our ability to generalize due to a small sample size, which seems counter-intuitive? $\endgroup$
    – M.JM
    Commented Mar 23, 2021 at 15:02
  • $\begingroup$ A small sample size affects precision, not generalizability. If the small sample resembles the population, its effect will generalize. There just may be uncertainty about what the effect is. Not all matching methods drop participants; full matching retains all participants and estimates the ATE. Template cardinality matching finds the largest matched sample possible given supplied target balance constraints. $\endgroup$
    – Noah
    Commented Mar 23, 2021 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.