Imagine I am running a survey with a non-probability sample. The population is 5,000 people, and my sample is about 100 people. There are two binary variables, $x_1$ and $x_2$, that predict the binary outcome of interest, $y$. In the population, each of the four combinations of $X$ are about 25% of the population. In the sample, we see non-response by $X$ and some groups are over-represented.
One way to address this is to rake across the margins of each $X$. I simulate this and then get an estimate of the proportion of people where $y = 1$. After doing it 5,000 times (R code for simulation here), I observe that the weighting works at correcting for non-response bias:
# A tibble: 4 x 2 type mean <chr> <dbl> 1 nonrand_sample 1.26 2 population 0.680 3 rand_sample 0.684 4 weighted_sample 0.684
These are the estimates taken from the formula
glm(y ~ 1, binomial, data, weights)
The non-random sample is far from the population estimate of $.68$, but the weighted sample matches the random sample (where there was no non-response bias by $X$).
However, we can also look at the distribution of these estimates:
The distribution of the weighted estimates is wider than from a random sample. We can look at the average standard error estimates from each of these groups, as well:
# A tibble: 3 x 2 type avg_se <chr> <dbl> 1 nonrand_sample 0.252 2 rand_sample 0.215 3 weighted_sample 0.221
The weighted sample has a larger standard error estimate than the random sample. Why does using weights increase our standard error estimate? It seems like, since the mean of the weights is 1, the effective sample size should not be increased or decreased. However, I get the intuitive explanation that there is uncertainty in our weights, so this should be folded into total uncertainty. But I do not see how that comes into play—mathematically speaking—when specifying weights in the