How can I use Propensity Scores to adjust for survey non-response bias?

Question

Say I estimate the probability that each member of my target population responds to a survey using propensity scores. I am having a hard time finding a clear explanation of how I can use the propensity scores to adjust a continuous outcome of interest, $y$. My current understanding is that I can compute a weighted mean of $y$ where the weights are the inverted propensity scores. Is that what is typically done? Or should the propensity scores somehow be leveraged in a regression model to estimate the population $\bar{y}$?

Answer 1

The answer to this depends on whether you have a probability sample versus a nonprobability sample, where a probability sample refers to a sample selected using random sampling from the population.

If you have a probability sample

For probability samples, you know each sample member's sampling selection probability (i.e. the probability that they would be asked to take the survey), the inverse of which is the basic survey weight. If you know the weights for both responding and non-responding members of the sample, then you would typically apply a non-response adjustment to the weights for the responding members of the sample. A common approach is response propensity class adjustment, where you divide your sample into, say, three groups based on the estimated response propensities: low, middle, and high. For each group, you would adjust the weights by multiplying by the factor N_hat_full / N_hat_responding, where N_hat_full denotes the sum of sampling weights for the full sample in that group and N_hat_responding denotes the sum of sampling weights for the responding members of the sample in that group.

This R package vignette demonstrates how to do this in R:

'svrep' package vignette on nonresponse adjustments

The method you're referring to--where you use the inverse of the propensity scores as weights--is inverse propensity score (IPS) weighting, which is not typically used as a non-response adjustment method for probability surveys. The method of propensity class adjustment is often preferred because it produces less variation in the weights and so is often preferable in terms of reducing variance of weighted estimates. Different methods for using propensity scores to adjust for nonresponse in probability surveys are explained and compared by Haziza and Lesage (2016).

If you have a non-probability sample

In the context of data from a non-probability sample, the inverse propensity score weighting (IPSW) approach is very commonly used to compensate for the fact that you don't know sampling probabilities since you didn't take a controlled sample of the population. Its use in this area was outlined and (I believe) first proposed by Lee and Valliant (2009); a good recent study discussing this approach for non-probability samples is a 2018 Pew Research report discussing different options for estimating the propensity scores.

In a nutshell, the process works as follows.

1. The data from a given sample of opt-in survey responses is combined with dataset from a synthetic population (which can be created using data from a probability sample). The observations from your survey are stacked on top of the observations from the synthetic population.
2. A model is trained to predict, for each record in the stacked dataset, whether that record came from the opt-in sample or from the reference dataset.
3. For each respondent to the opt-in survey, the trained model is used to generate a probabilistic prediction for whether that respondent’s row in the stacked dataset came from the synthetic population rather than the opt-in sample.
4. An inverse propensity weight is calculated for each respondent as w_i = p_i/(1−p_i), where p_i denotes the predicted probability that the respondent’s record was drawn from the reference dataset rather than the opt-in sample.
5. (Optional) Weights are rescaled to match the total population size

The weights are then used as if they were simple sampling weights: population totals are calculated as SUM(WEIGHT * X); means are typically calculated as the SUM(WEIGHT * X) / SUM(WEIGHT).

References

• Haziza and Lesage (2016). A Discussion of Weighting Procedures for Unit Nonresponse. Journal of Official Statistics, 32(1) 129-145. https://doi.org/10.1515/jos-2016-0006

• Lee and Valliant (2009). Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment. https://journals.sagepub.com/doi/abs/10.1177/0049124108329643

• Mercer, Andrew, Arnold Lau, and Courtney Kennedy. 2018. “For Weighting Online Opt-in Samples, What Matters Most?” Pew Research Center. https://www.pewresearch.org/methods/2018/01/26/for-weightingonline-opt-in-samples-what-matters-most/.

• Valliant, R., Dever, J., Kreuter, F. (2018). Practical Tools for Designing and Weighting Survey Samples, 2nd edition. New York: Springer. https://doi.org/10.1007/978-3-319-93632-1. Chapter 13 provides an excellent overview of nonresponse adjustment methods for probability surveys, and chapter 18 provides an overview of weighting methods for nonprobability surveys.