I recognize that propensity scores are often used for causal inference. Just to clarify from the outset, that's not what I'm interested in here.

Instead, I'm looking at using propensity scores to match observations from a probability sample with observations from a nonprobability sample so that I can assign weights to the nonprobability sample in the hopes that the re-weighted nonprobability sample will more closely represent the target population. As I understand it, this is relatively common practices for web surveys, etc. and there's extensive literature on the topic.

Where I'm confused is trying to get my head around how to use the survey design weights from the probability sample in the calculation of propensity scores. This article by Valliant and Dever (2011) discusses the issue of weights but I don't entirely follow the conversation about weights.

Could anyone explain what I need to do here and perhaps also how to do it in R, preferably with the MatchIt package? Here are a few of my thoughts.

If I used something like nearest neighbor, I could just match the observations and then following the advice in the article above under "Mechanics of Estimating Propensity Scores,"(b) by dividing the observations into quantiles of equal size (utilizing propensity scores) and then weight the probabilistic observations in each quantile by taking the average probability for all the observations in the quantile.

However, this would ignore the survey weights of the probability sample. I could perhaps utilize those weights after the fixing the quantiles by taking the average survey weights for each probability observation in quantile and multiplying this by the initial weight for the quantile. Probably doesn't make sense but I'm just thinking out loud at this point.

Perhaps the MatchIt package or another package in R facilitates the utilization of survey weights in the calculation of the propensity scores (and not just by adding the weights in as a variable).

Any direction would be most appreciated!


1 Answer 1


You don't need to use matching to do this. Ideally, you want to estimate new survey weights that when applied to the nonprobability sample make it look similar to the survey-weighted probability sample. This is analogous to using propensity score weighting to reweight a control group so that it resembles a treated group, except in this case the treated group (i.e., the probability sample) already has weights. The R package WeightIt can help you estimate weights in this scenario.

You'll want to structure your data so that you have a variable that indicates whether each observation comes from the probability (1) or nonprobability sample (0), and another variable that represents the existing sample weights (the survey weights for the probability sample and 1 for the nonprobability sample). Then, you want to estimate ATT weights, which will reweight the nonprobability sample to resemble the survey-weighted probability sample. There are a variety of methods to do so. Ackerman et al. (2020) found that generalized boosted modeling with balance targeting did the best of the four methods they considered, but you can try other methods as well. Once you've estimated the weights, you can assess the degree to which the samples resemble each other. This is analogous to checking for covariate balance in observational studies. You can use the cobalt package to do this.

Below is some sample code that you could use (replacing the variables with your own). I'll use S for the probability/nonprobability sample indicator (with S = 1 for the probability sample), and sw for the survey weights, which, again, are equal to 1 for the nonprobability sample and equal to the survey weights for the probability sample.

library(WeightIt); library(cobalt)
w.fit <- weightit(S ~ X1 + X2 + X3, data = combined_data,
                  s.weights = "sw", estimand = "ATT",
                  method = "ps")

If you just want to use the newly weighted nonprobability sample and ignore the probability sample, you can extract the weights from the w.fit object and subset those with S == 0. If you want to use both samples combined, you need to multiply the estimated weights by the original survey weights (i.e., new_w <- combined_data$sw * w.fit$weights).

In the call to WeightIt, you can specify different estimation methods with the method argument. "ps" is logistic regression-based propensity score weighting. You can try "gbm" or "super" to use the methods in Ackerman et al. (2020), who also use WeightIt in their simulation, but these methods require additional choices. "ebal" or "optweights" can be good choices because they guarantee exact balance on the means of the covariates you include, but they require additional assumptions about the form of the selection model.

Note: I'm the author of both WeightIt and cobalt.

  • $\begingroup$ Noah, thank you very much! I'll take a look at all of this. $\endgroup$
    – num_39
    Commented Apr 22, 2020 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.