2
$\begingroup$

Imagine we conduct a survey using stratified sampling, and after the survey closes, we find out that we had misclassified some of the respondents. They actually belong to different strata than we had assigned.

We can let $n_H$ and $N_H$ be the size of stratum H in our sample and in the population, respectively. $n^*_H$ and $N^*_H$ are the corrected values after we adjust membership using information we learned when reading the responses.

How do we calculate weights and assign strata in this case? Do we leave the original strata unchanged, leave the mis-assigned respondents in their original strata, and use $n_H$ and $N_H$ to calculate base weights?

Or, do we make an adjustment to the strata/weights and re-assign respondents to the correct strata? How are base weights calculated in this case--are they simply $N^*_H/n^*_H$? This seems wrong because the weights are no longer 1/selection probability, as the selection probabilities we used to draw the sample are $n_H/N_H$.

Maybe there's a third option: we do not change the base weights, and we leave them as $N_H/n_H$ (i.e., 1/selection probability), and we simply change the stratum assignment of those respondents who were misclassified? But this seems wrong too, since it creates different weights within a single stratum, which I thought raised issues.

$\endgroup$

1 Answer 1

1
$\begingroup$

I found the answer to my question. tldr; respondents should not be reassigned to different strata after sampling has occurred, and there's no need because "misassigned" respondents do not negatively impact representativeness. The sample is still representative as long as the design weights are calculated from the selection probabilities.

The point of stratification is to draw samples from groups that are similar and thus have lower within-group variance. The underlying assumption is that the stratifying variables partition the population into these lower-variance groups, but (a) that may not always be true, (b) some variables may be better/worse at generating lower-variance groups, and (c) the stratifying variables themselves are arbitrary and don't get interpreted when reporting results. The fact that some respondents were later found to be misassigned simply means (a) the strata are (maybe) less homogeneous than we hoped and thus have slightly higher within-group variance, and (b) the stratifying variable cannot be interpreted in the way it was originally conceived.

Here's a concrete example. Two statisticians are administering a survey on health in a city. They purchase data from a market research firm with demographic and contact info on every citizen. One statistician stratifies the population by geography, age, and political party. The other statistician thinks health doesn't differ across political party and only stratifies by geography and age.

They both draw their sample and administer the survey. One of the questions on the survey is "what is your political party?" The first statistician reads the answers and learns that the sampling frame they purchased had some incorrect data, and some of the respondents have different political parties than originally thought. Consequently, the first statistician's strata are somewhat mixed on political party. But the fact that the political parties are mixed together doesn't impact representativeness. After all, the second statistician's strata are completely mixed on political party. Both samples are still representative of the population as long as the design weights are calculated properly based on the original selection probabilities.

This issue only impacts interpretation. If the first statistician writes a report on the survey results, it would be somewhat incorrect to state that the sample was stratified by political party. A better statement would be that the sample was stratified by "estimated" political party, with a description of how the political party was estimated / where that demographic data came from. The statistician could even call the purchased data the "estimated political party," which differed from the "true political party" that respondents reported in their survey.

If the statisticians want to do subgroup/domain analysis and analyze health by political party, they they should both use the true political party variable, not the estimated variable. This way, they will be correctly representing the data when they say "On average, health is X within political party A."

On the other hand, if the statisticians want to do a nonresponse bias analysis, then they should use the estimated variables (assuming that's the only data they have on nonrespondents). When they describe the analysis in their report, they should explain that they used the estimated political party to conduct the nonresponse bias analysis, because the estimated variables were available for every member in the population. However, the estimated variables do not correspond perfectly to the true political party and have known inaccuracies.

It helped me to realize that, despite its inaccuracies, there might still be valid ways to interpret the original estimated stratifying variable. For example, maybe the market research company estimated citizens' political party by looking up their parents' political party, which was perfectly known. This would explain the discrepancies, since some people choose a different party from their parents. But it also shows how it's possible for the original "inaccurate" variable (estimated political party) to be a better choice for stratification than the "correct" variable (true political party). After all, a person's health can be heavily influenced by their parents' habits, so if political party is relevant to health, then the parents' party might be more indicative than the person's own party.

(I have no idea if political party actually has anything to do with healthy; I'm using it purely as an example of a variable for which purchased sample frame data might conflict with survey responses.)

$\endgroup$
1
  • 1
    $\begingroup$ Yes, this answer is correct. For the purpose of weighting and variance estimation, you should use the strata that were actually used in sampling. For reporting estimates, you can use information that you learned after sampling, but you just need to make sure that the underlying weights and standard errors are based on the actual sampling variables. $\endgroup$
    – bschneidr
    Feb 15, 2023 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.