# Evaluation of Propensity Score Matching: quantify bias of variation in sample distribution

I've completed propensity score matching of a treatment and control group across a number of covariates. On two categorical covariates we require an exact match, for example, Gender and Eye Color, and we're trying to determine effect of a treatment on continuous feature like salary (silly example).

However, there is one group of, say females with brown eyes, that are under represented in the unmatched data. Because of the exact matching, this causes males with brown eyes to be under represented in the resulting matched data (regardless of their significant representation in the unmatched data). I'm trying to determine how this could be impacting our salary prediction and quantifying the potential error due to bias in the matched sample.

Put another way, what is the potential error in the matched data due to differences in covariate distribution between matched and unmatched samples?

I've gone down the path of assessing confounding variables and effect modification, but I'm not sure 1) if I can compare those effects across populations and 2) how I should present the information [1]. I might be over complicating, but it would seem that I need to adjust for the confounding features and than adjust the effect modification by the distribution in the matched data? I've also tried estimating treatment effects after matching [2], bootstrapping, and rbounds [3] but none seem to be addressing what I need-- I suspect that I'm just phrasing the question wrong or missing the obvious?

The implication is that pair matching cannot be used to estimate the ATE. It sounds like you want to estimate the ATE since you are worried that your matched sample is not representative of the unmatched sample, the latter of which is (ideally) representative of your population of interest. If you want to estimate the ATE, you can just use one of the many methods available that do estimate the ATE. Pair matching (as implemented in MatchIt) is not one of them. Among matching methods, you can use full matching, propensity score subclassification, or template cardinality matching. You can also use weighting methods like inverse probability weighting that target the ATE. You can also use matching imputation as implemented in the Matching package and teffects in Stata.