# How can I use Propensity Scores to adjust for survey non-response bias?

Say I estimate the probability that each member of my target population responds to a survey using propensity scores. I am having a hard time finding a clear explanation of how I can use the propensity scores to adjust a continuous outcome of interest, $$y$$. My current understanding is that I can compute a weighted mean of $$y$$ where the weights are the inverted propensity scores. Is that what is typically done? Or should the propensity scores somehow be leveraged in a regression model to estimate the population $$\bar{y}$$?

## 1 Answer

### If you have a probability sample

For probability samples, you know each sample member's sampling selection probability (i.e. the probability that they would be asked to take the survey), the inverse of which is the basic survey weight. If you know the weights for both responding and non-responding members of the sample, then you would typically apply a non-response adjustment to the weights for the responding members of the sample. A common approach is response propensity class adjustment, where you divide your sample into, say, three groups based on the estimated response propensities: low, middle, and high. For each group, you would adjust the weights by multiplying by the factor N_hat_full / N_hat_responding, where N_hat_full denotes the sum of sampling weights for the full sample in that group and N_hat_responding denotes the sum of sampling weights for the responding members of the sample in that group.

The method you're referring to--where you use the inverse propensity scores as weights--is inverse propensity score (IPS) weighting, which is not typically used as a non-response adjustment method for probability surveys.

### If you have a non-probability sample

In the context of data from a non-probability sample, the inverse propensity score weighting (IPSW) approach is very commonly used to compensate for the fact that you don't know sampling probabilities since you didn't take a controlled sample of the population. Its use in this area was outlined and (I believe) first proposed by Lee and Valliant (2009); a good recent study discussing this approach for non-probability samples is a 2018 Pew Research report discussing different options for estimating the propensity scores.

In a nutshell, the process works as follows.

1. The data from a given sample of opt-in survey responses is combined with dataset from a synthetic population (which can be created using data from a probability sample). The observations from your survey are stacked on top of the observations from the synthetic population.
2. A model is trained to predict, for each record in the stacked dataset, whether that record came from the opt-in sample or from the reference dataset.
3. For each respondent to the opt-in survey, the trained model is used to generate a probabilistic prediction for whether that respondent’s row in the stacked dataset came from the synthetic population rather than the opt-in sample.
4. An inverse propensity weight is calculated for each respondent as w_i = p_i/(1−p_i), where p_i denotes the predicted probability that the respondent’s record was drawn from the reference dataset rather than the opt-in sample.
5. (Optional) Weights are rescaled to match the total population size

The weights are then used as if they were simple sampling weights: population totals are calculated as SUM(WEIGHT * X); means are typically calculated as the SUM(WEIGHT * X) / SUM(WEIGHT).

References

• Thank you. This is exactly what I was describing except calculating the weights as (1/ 1-p) as opposed to 1/p. Your final sentence (after #5) is the part that I've been wondering about.
– Alex
Mar 15, 2021 at 18:12