Sample survey: can I weight back to the target population from the survey population?

Question

I am working for an organization that regularly polls its members. In a previous study, the researchers started with a target population frame of 50,000 people. They then eliminated 15,000 on the grounds that they had recently received other surveys, leaving a survey population of 35,000. From this, they drew a stratified sample of 4,500 people. 1,730 completed surveys were returned.

The researchers stratified on the basis of the 35,000 and calculated survey weights on that basis. However, they seem to have adjusted the weights to give results for the 50,000 --- the sample weights add up to 50,000. They also did some non-response weighting, based on the observation that proportionately more women than men responded. These weights were based on stratum totals from the sample of 4,500 and the 1,730.

My questions are:

1. Is it OK to weight back to the original, target population?
2. If so, what should the weights be?
3. What happens to the variance estimates?

Assume that we are interested in estimating a population total.

Note that the survey population of 35,000 is not a simple random sample of the 50,000. It is the result of removing several stratified samples from the 50,000, with non-proportional strata.

Answer 1

1. Is it OK to weight back to the original, target population?

As a general rule, yes, it is okay, and indeed desirable, to weight back to the original target population. Your goal in these problems is usually to estimate an unknown population quantities that is aggregated over a stratified group. If the numbers of people in each group in the population is known (e.g., known number of males and females) then it is generally a good idea to weight the sample estimators in such a way that they account for the known sizes of the population groups. In this particular case, it may be dubious to make inference beyond the sampling frame of 35,000 people into the broader population of 50,000, but that is a separate issue.

2. If so, what should the weights be?

3. What happens to the variance estimates?

It sounds like you have a complex sampling problem, so this is a complex question that would need to be considered in light of a detailed understanding of the sampling scheme and estimation methods. However, to give you an idea of the principles involved, I will give a simpler example of a stratified sampling problem with known sizes for the population groups.

Consider the case where you have a population of size $N = N_M + N_F$ consisting of $N_M$ males and $N_F$ females. Each person has some characteristic quantified by a variable $X_i$ and you want to make inferences about the population mean $\bar{X}_N$. Suppose you sample from this population using stratified random sampling with $n_M$ males and $n_F$ females. You obtain sample means $\bar{X}_M$ and $\bar{X}_F$ for these two groups. In this case your estimator of the population mean would be:

$$\hat{\bar{X}}_N = \frac{N_M}{N_M+N_F} \cdot \bar{X}_M + \frac{N_F}{N_M+N_F} \cdot \bar{X}_F.$$

We can examine this estimator under the superpopulation approach, where the finite population is embedded in a larger model with mean and variance parameters. Under this approach it can be shown that:

$$\begin{equation} \begin{aligned} \mathbb{E}(\hat{\bar{X}}_N - \bar{X}_N) &= 0 \\[10pt] \mathbb{V}(\hat{\bar{X}}_N - \bar{X}_N) &= \frac{1}{(N_M+N_F)^2} \Bigg[ \frac{N_M (N_M - n_M)}{n_M} \cdot \sigma_M^2 + \frac{N_F (N_F - n_F)}{n_F} \cdot \sigma_F^2 \Bigg]. \end{aligned} \end{equation}$$

This gives you the quasi-pivotal quantity:

$$T = \frac{(N_M+N_F) \cdot (\hat{\bar{X}}_N - \bar{X}_N)}{\sqrt{N_M (N_M - n_M) S_M^2 / n_M + N_F (N_F - n_F) S_F^2 / n_F}} \overset{\text{Approx}}{\sim} \text{T-Dist}(DF),$$

where the degrees-of-freedom $DF$ are found using the Welch-Satterthwaite method. As you can see, the variance of the difference $\hat{\bar{X}}_N - \bar{X}_N$ is affected by the weighting in the estimator. Given a prior assumption about $\sigma_M^2$ and $\sigma_F^2$, minimisation of this variance subject to the constraint $n = n_M+n_F$ can be used as an optimisation problem to find the optimal sample sizes for the strata.

Answer 2

It's not really clear to me why 15,000 potential cases were removed from the original sampling frame. But assuming there was good reason and assuming those 15,000 people never had a chance to respond to the survey, the survey weights should add to 35,000, not 50,000. Perhaps the original analysts did not want to bother 15,000 members again with another survey since they recently returned one so they excluded these 15,000 people from being eligible to participate in this survey based on the critical assumption that those 15,000 do not differ in any meaningful way from the other 35,000 people that were left in the original sample frame. If that was the case, then you could proceed as follows with only minor "hand-waving":

1. Initial sampling fractions $f_h={n_h\over{N_n}}$within each of the $h$ strata are determined by taking the total number of those sampled in a given stratum $n_h$ and dividing by the total number of people in the population/sampling frame in stratum $h$, $N_h$. The base weight in stratum $h$ at this point is calculated by taking ${1\over{f_h}}={N_h\over{n_h}} \equiv w_h$. The $i$th respondent in stratum $h$ will then be assigned this weight and we'll call it $w_{ih}$.

2. If there were no such thing as non-response, the sum of the weights $w_{ih}$ would add up 50,000, but if you have missing data, then that might not be the case. Then you'd want to weight for non-response. One method of doing this - and it sounds like maybe the method your organization used in this case - is to use stratified weight-class adjustment weights based on gender. Here is how this works. Within each stratum separate your sample by men and women. For the men, add up all their sample weights and divide this by the sum of all the sample weights for the male respondents. Do the same thing for women and the female non-respondents. These are then your Weighting Class Adjusted non-response weights, $\pi_{ihc}$ for person $i$ in stratum $h$, gender-class $c$.

3. You can now find your final weight, $\phi_{ih}$, for person $i$ in stratum $h$ gender-class $c$ by multiplying $w_{ih}\pi_{ihc}=\phi_{ihc}.$ The sum of all these weights should now equal 50,000 (maybe minus some small rounding error).

Generally speaking your variance will increase when you weight for non-response due to an increase in variability caused by the fluctuating values of the weights. The variance estimates can be calculated most easily by any software that is capable of analyzing weighted survey data (e.g. SUDAAN, SAS, SPSS Complex Samples, etc.). The formulas for calculating this can get pretty complex, so I won't write them out here and this isn't something you really want to do by hand. Instead, use one of the survey packages for the analysis of complex sample surveys and set up the appropriate design and weighting variables.

The only way your results are generalizable to the enter population of 50,000 is if one can assume the 15,000 that were excluded from the survey are not systematically different in any way measured by your survey. It sounds like the 15,000 cases might have been removed by removing all those sampled in specific strata. If you there is any chance those those missing strata are somehow different from your other strata along some dimension you are measuring in your survey, then you may not generalize your findings to all 50,000, but to only those strata from which you sampled.

Answer 3

It sounds like your researchers already gave you a weight to the original target population. My guess is that you're using a program like SPSS, and you want to use the weight given to you by these researchers, but you want your observations to sum to 1,730 and not to 50,000. That's not unusual at all. This is what is often called 'normalizing' your weights. To do this, just divide each weight by the average weight. The new weight will sum to 1,730.

To your three specific questions:

1. Yes, it's okay. But you're not "weighting back to the original target population." You're just scaling the same weight (i.e. using a different denominator). If you want to go back to the original sample (without the weights), then just turn them off.

2. Divide each individual weight by the average weight for the total sample. If you're in SPSS, do this by going to Data -> Aggregate.

3. If you're working with percent estimates, nothing much will change. The caveat is that if you have a design effect greater than 1 (you probably do), then your "effective sample size" is probably less than 1,730. But to work this out, you'd have to go back to whoever did the original weighting for you; there's not enough information here for me to work this out. What it means is that you're probably going to underestimate your variance with whatever weight you use (normed or otherwise), unless you're using a complex samples design (like with the 'complex samples module' in SPSS or the 'survey' package in R).

And yes, you're results are still generalizable to the total population even after you've rescaled your weight. Estimate the percent of people who drink Pepsi, multiply this by 50,000 and that is the number of people in the total population who drink Pepsi.

Answer 4

Do you mean target "sample" instead of target "population". A sample is defined as a subset of a population. The weights for a complex sample are often interpreted as the number of people in the same "strata" in the population for a given case (i.e., with the same demographic variables such as geographic region, age, sex, etc.). For example, if the weight for a given case in a sample is 50, then there are 50 people just like that person in the population on key demographic variables.

When you apply weighting of cases to a complex sample, you are projecting the results for the entire population.

This is a problem for significance testing, as you are spuriously inflating the number of cases to the population size, and not the sample size.

To correct for this, you need to multiply the weights by a constant (which will have no ill effect on the estimation of proportions and means because it is a constant).

The NEW_WGT=OLD_WGT * sample_size /population_size. The sample size should be readily apparent (i.e., number of cases), and the population size is the sum of the old_wgts. The new_wgts are sometimes called "normalized" weights.

Now you will have the correct proportions and means, and the correct original sample size for significance testing.

However variances, covariances, and standard errors are messed up when you weight cases.

Now you have to adjust for the stratification and design effect in your complex sample, which is a whole other discussion.