How to match a biased sample to a population?

Question

I have a sample of people which is biased in age, gender, geography. I am trying to measure various continuous outcomes out of them. I have the census data to tell me the reality of the population distribution at the zipcode level/county/state level and I cannot change my biased sample. How can I alter my sample to be a good representation of the population?

I guess I would like to have a set of weights for each one of the people in my sample, to alter the outcome according to these weights, determined by the mismatch between my sample and the population. These weights should be normalized to 1 to make sure the full aggregated outcome remains the same.

Another idea would be to sub-sample to match the population distribution, but I would potentially lose a lot of data so I'd rather not go that way.

I started reading about the Heckman correction. Is it the place to start? Is there a standard method which can help me achieve that goal? Do you know of any good book that treats this question?

Answer 1

The Heckman model is for a different problem. You were right to look at weighting. Your problem looks very suitable for raking, see e.g. https://www.stata-journal.com/article.html?article=st0323

Answer 2

The simplest thing you can do (while not necessarily the efficient one given your purpose) is to define cells based on age-gender-geography. See what is the share of people of the population belonging to that cell and compare it to the share of people in the sample that belongs to that cell. The ratio of the former to the latter would give you the corresponding weight for each observation (which is constant for observations within a cell)

But the more fundamental problem is that because you don't have a balance on observables, you most likely don't have a balance on unobservables as well. This implies even if you re-weight your sample to be representative of the population wrt observable characteristics, it's probably not a good representative of the population on things you cannot directly measure.

Answer 3

The quick way to derive good weights on pencil and paper is via the Horvitz-Thompson estimator. Each weight is just the reciprocal of the probability of sampling each instance. However, you can more easily weigh your samples with the likelihood ratio of their occurrence in the real population vs in your sample $\frac{\mathrm{P(\mathrm{zip},\mathrm{county},\mathrm{state},\mathrm{gender},\mathrm{age}|\mathrm{population})}}{P(\mathrm{zip},\mathrm{county},\mathrm{state},\mathrm{gender},\mathrm{age}|\mathrm{sample})}$. In your case zip supersedes county supersedes state. The latter is particularly straightforward within typical dataframe software. You can find a tutorial involving a similar scenario with a derivation here. Note neither of these weights sum to 1, but they are theoretically sound.

Answer 4

This problem has becomes of interest lately in epidemiology, so the following answer is from that point of view.

First, we need to distinguish where the sample originates from. If the biased sample comes from the population (i.e., it is nested in the population), then these problems have been referred to as 'generalizability'. If the sample comes from a different population (e.g., we are using data from Ohio but the population of interest is North Carolina), then these problems are referred to as 'transportability'. This distinction is important, since it will influence what estimators are selected.

Generalizability

In order to generalize results from the biased sample to the target population, we need to make a few (unverifiable) assumptions. Namely, we will stipulate that the outcomes and selection into the sample are independent (exchangeable). Generally, we would make some conditional exchangeability statement. For example, you may choose (possibly incorrectly) to assume that selection and all your various outcomes are independent given age, gender, and geography. Along with this assumption, all the values of the covariates for exchangeability that are in the target population need to be seen in the biased sample.

For estimation, there are a few options. As you suggested, you could choose to re-weight the biased sample to 'stand-in' for the target population. These are sometimes called inverse probability of selection weights (IPSW). However, you could also fit a model in the biased sample with the continuous outcome modeled by the factors for exchangeability. Then the estimated coefficients could be used to predict the outcome values in the target population data. Both of these approaches (and the identification conditions) are detailed in this excellent paper. There is also augmented inverse probability of selection weighting, which is doubly robust (description of that estimator can be found here).

Transportability

Transportability relies on related identification conditions. Additionally, there are similar adaptions of the estimators. However, there are some important differences. For example, the weighting approach now uses inverse odds of selection weights. Details on the weighting estimator can be found here and other related estimators can be found here.

Note: most of these estimators talk about generalizing or transporting causal effects, but your problem can just be viewed as a one-sample problem (a simplification of most of the problems consider in the above papers). This means that the assumptions and adjustment methods are treatment / exposure are extraneous to your application.