Proper subsetting of survey data

Question

I have a question about proper subsetting of survey data. This is the scenario: A health survey targeting the general U.S. population has a first stage of sampling that begins by defining primary sampling units (PSUs) as counties, or groups of counties for sparsely-populated areas. These PSUs are stratified by state (Washington, DC is lumped in with Maryland), and two are selected within each stratum, for a total of 100 sampled PSUs. Thereafter, subsequent stages of sampling are done in a hierarchical manner down to the household level and, ultimately, one individual within the household is selected to participate in the survey. For variance estimation purposes, however, the survey administration team only releases stratum and PSU codes.

If you were estimating the rates of a particular health condition (e.g., Type 2 diabetes) captured by the survey for a region of the country, which you defined as a grouping of 8 states, could you subset the survey data set for all cases in the region?

I'm confused about whether this would be a proper subset of the data if subsetting as described above. I believe that it is not because the number of people with a health condition could vary from one county to the next in the sample, so the domain sample size is a random variable itself. However, I'm not totally positive since it seems like a logical way to subset your data.

Answer 1

Here's a well-written blog post that directly answers your question: https://notstatschat.rbind.io/2021/07/22/subsets-and-subpopulations-in-survey-inference/

Short answer:

The short answer is that, no, you don't want to simply subset your data and calculate the estimate from the subset. The variance of the estimate from your subset depends on the entire design, not just the sample members from the domain you're interested in. The only practical exception is that if your data has replicate weights created using the complete data, you can proceed to estimate variances for domain estimates by just subsetting the data and using only the rows of replicate weights in your subset of data. But again, the replicate weights have to first be created using the full dataset.

Comment on survey statistics software

To provide a user-friendly experience, survey statistics software will "pretend" to subset the data, but what it actually does is retain all the data but set the weights to zero for cases outside the subset of interest. For example, consider the following R code adapted from Dr. Lumley's blog post, which estimates the total enrollment for high schools (i.e. ignoring middle and elementary schools):

library(survey)

data(api)

# Create a survey design object
dclus1 <- svydesign(id=~dnum,weights=~pw, data=apiclus1, fpc=~fpc)

svytotal(~enroll, subset(dclus1, stype=="H"))

What this code is actually doing is equivalent to the following:

svytotal(~I(enroll * ifelse(stype=="H", 1, 0)), dclus1)

In the above code, R is keeping track of the entire survey design, even when you're computing estimates for just a subpopulation. You'd run into problems if you didn't tell R about the entire survey design, for example if you did the following:

# Don't do this!
library(survey)

data(api)

subset_of_data <- subset(apiclus1, stype == "H")

# Create a survey design object using only the subset
dclus1 <- svydesign(id=~dnum,weights=~pw, data=subset_of_data, fpc=~fpc)

svytotal(~enroll, subset(dclus1, stype=="H"))
```