Matching a Subsample with Complex Survey Data

Question

I am completing a study to analyze the effects of a policy treatment on a sub-population of adults (low-income adults, specifically) that are represented in a complex, nationally representative survey.

I want to conduct a matching procedure that balances members of this sub-population with respect to treatment (1=yes; 0=no). It is unclear to me, however, whether I should restrict my matching procedure to include only members of the sub-population of interest (i.e. low-income adults), or whether my matching procedures should include all individuals in the survey population and simply include income as a match variable.

I'm leaning towards the latter option (i.e. include the entire sample in matching) because I know that my matching weights need to be multiplied by the sampling weights to complete my subsequent outcome analysis, and existing software easily facilitates this transformation. However, it seems somewhat silly to include all members of the overall survey sample in my matching procedure, as the treatment of interest is only relevant to people with low-incomes. Therefore, my matching procedure would be including numerous irrelevant observations.

My question here is as follows: is it bad statistical practice to conduct my matching procedure on the overall survey population even though the treatment of interest is only relevant to a small fraction of that population?

Answer 1

The point of matching is to estimate a counterfactual mean, i.e., what would have happened to the treated units had they not been treated. If income is related to the outcome (and surely it is, or else the policy would not have targeted low-income people), then the distribution of incomes must be the same in the treated and matched control group for your effect estimate to be unbiased. I think it's clear that not restricting your sample to low-income people will potentially allow high-income people to enter your matched sample, which would prevent the distributions from being the same.

You can do whatever you want to match your sample, but you should evaluate balance on income (and all other variables, of course). You should try several matching methods to see which yields the best balance. Propensity score matching with income as a covariate in the propensity score model will probably not work well at balancing income (because it's possible for two units to have similar propensity scores and very different values of income), whereas Mahalanobis distance matching or any form of matching with a caliper on income could balance income.

So, my answer is the same as it is in most matching questions: try many methods and use the one that yields the best balance. There is no theoretical guidance for how any specific method will fare in your dataset. I guarantee that restricting your control sample to a sample with similar income values as the treated units will make it easier to get balance on income, though.