Imagine I am running a survey with a non-probability sample. The population is 5,000 people, and my sample is about 100 people. There are two binary variables, $x_1$ and $x_2$, that predict the binary outcome of interest, $y$. In the population, each of the four combinations of $X$ are about 25% of the population. In the sample, we see non-response by $X$ and some groups are over-represented.

One way to address this is to rake across the margins of each $X$. I simulate this and then get an estimate of the proportion of people where $y = 1$. After doing it 5,000 times (R code for simulation here), I observe that the weighting works at correcting for non-response bias:

# A tibble: 4 x 2
  type             mean
  <chr>           <dbl>
1 nonrand_sample  1.26 
2 population      0.680
3 rand_sample     0.684
4 weighted_sample 0.684

These are the estimates taken from the formula glm(y ~ 1, binomial, data, weights)

The non-random sample is far from the population estimate of $.68$, but the weighted sample matches the random sample (where there was no non-response bias by $X$).

However, we can also look at the distribution of these estimates:

enter image description here

The distribution of the weighted estimates is wider than from a random sample. We can look at the average standard error estimates from each of these groups, as well:

# A tibble: 3 x 2
  type            avg_se
  <chr>            <dbl>
1 nonrand_sample   0.252
2 rand_sample      0.215
3 weighted_sample  0.221

The weighted sample has a larger standard error estimate than the random sample. Why does using weights increase our standard error estimate? It seems like, since the mean of the weights is 1, the effective sample size should not be increased or decreased. However, I get the intuitive explanation that there is uncertainty in our weights, so this should be folded into total uncertainty. But I do not see how that comes into play—mathematically speaking—when specifying weights in the glm() command.


I did more research and answered this question with a white paper.


We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.