8
$\begingroup$

We surveyed all 10,000 professionals in a particular industry. The industry is highly-regulated, so we have contact information for everyone in our population of interest. We attempted to contact 100% of the population. We now have a data set containing the 2,000 responses, because 20% of our population agreed to complete the survey. When administering this survey, there were no sampling probabilities and no clustering at all.

There is a lot of variation in the response rate when broken out by state of residence. Since state variation is important in this industry, we plan to calculate weights for this final data set so that any statistics that we run will generalize to the overall population rather than the 20% who responded. I believe the weights should be treated as post-stratification weights, but I'm not certain.

I don't imagine this is a terribly complex data set to analyze, but I'm not sure if it's a special case of some sort -- it didn't involve any sampling whatsoever but at the same time it is not the entire universe.

I would appreciate any coding tips (in any statistical language) to recommend the survey analysis setup that makes the most sense for data of this structure.

if I had to guess, here's the R code I would use:

# start with data set `x` and add a column of five, since 20% responded
x$wgt <- 5
# give everyone in the data set a weight of five

# provide only a column of 5's to the `svydesign` command
y <- svydesign( ~1 , data = x , weights = ~ wgt )

# create a table with the intended joint distribution, here with just two example states
pop.types <- data.frame( state = c( "state 1" , "state 2" ) , Freq = c( 5000 , 5000 ) )

# create the post-stratified survey design
z <- postStratify( y , ~ state , pop.types )

# have fun running statistics and confidence intervals
svymean( ~ variable.to.analyze , z )
confint( svymean( ~ variable.to.analyze , z ) )

ucla has a post-stratification tutorial in stata that makes me think it might be smarter to create the svyset line like this--

gen total_pop = 10000
gen pststr_wgt = .
replace pststr_wgt = 5000 if state == "state 1":state
replace pststr_wgt = 5000 if state == "state 2":state
svyset _n , fpc( total_pop ) poststrata( state ) postweight( pststr_wgt )
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ These data are worthless for making inferences about the population. You will have to be content with summarizing what you have found and limiting your interpretations to the population of professionals in your industry who answered your survey. But that doesn't need any statistical modeling; its a matter of summary statistics and graphics. Your alternative is to perform follow-up sampling to capture some of the non-respondents (in a representative way) in order to assess how respondents might differ from non-respondents. $\endgroup$
    – whuber ♦
    Jun 21, 2014 at 12:58
  • 1
    $\begingroup$ @whuber - what response rate would you need to regard the data as not worthless to make inferences? I have a feeling with such a statement you would flippantly dismiss very large swaths of survey research (which is fine, but is a very tough standard to accomplish anything). A 20% response rate is not abnormally low for a large proportion of survey research. $\endgroup$
    – Andy W
    Jun 22, 2014 at 16:42
  • 2
    $\begingroup$ @Andy The answer depends on the application. For instance, a survey (like this one) intended to characterize small subpopulations (which might selectively decide not to respond) would need very close to 100% participation. (This is the basic problem encountered by the US Census Bureau.) The only reason one could justify a 20% response rate would be with evidence (such as past experience) that the nonrespondents do not differ from the respondents. Famous examples exist of what can go wrong otherwise even with high response rates. $\endgroup$
    – whuber ♦
    Jun 22, 2014 at 17:27

2 Answers 2

Reset to default
4
$\begingroup$

Non-response is a frequent problem in surveys, to be sure. Any use of these data would need to include some language about the non-response rate, and the results judged with caution.

As to your question about R usage, the code looks fine. As a detail, instead of using weights in the svydesign part, I would use fpc=~rep(10000,10000). If you supply weights, you don't get the finite population correction -- just a weighted estimate.

Post-stratification is not the answer to all of life's problems. It can reduce the variance of the estimates, but not as much as if you had stratified from the start. And of course, it does nothing for you if the variable of interest is unrelated to the post-stratification variable, and related to whatever caused the non-response.

I think that a place to start in this case is to see whether the distribution of states in your responders is typical of your professional group. If it is, then post-stratification is unnecessary. If you know the age and gender of your people, you could test for those as well.

Post-stratification does literally nothing if you are calculating means of the post-stratification groups. (Example: I post-stratify on gender and then calculate the means for men and women). This is due to a design decision by Thomas Lumley, which he discusses on page 137 of his book, Complex Surveys: a Guide to Analysis using R.

I do not believe that your situation is as dire as @whuber believes, nor does it compare to the Literary Digest snafu during the 1936 presidential election. The LD drew a sample from a population frame that favoured the Republicans. They would have had problems even with a good response rate. Your population frame is complete.

Remember that you actually did, physically, receive information from 2,000 out of 10,000 people. You saw what you saw. The missing 8,000 would have to be very different from the 2,000 to massively shift your estimated mean from where it currently is. How bad can it be? Do some sensitivity tests and publish your results, I say.

Improve this answer
$\endgroup$
1
$\begingroup$

What do you mean by "there were no sampling probabilities" ? Did all units have the same inclusion probability (simple random sampling) ? Or is it because you tried to contact every unit in your population (in which case all initial weights are equal to 1) ?

Anyway, your results can clearly be used (even though your response rate is a little low, estimations on very small sub-populations might not be very precise) ! You'll be fine as long as you account for non-response in your estimations.

Post-stratification is one way to do so (then your post-stratification variables have to be highly correlated with non-response probability), but I'd recommend using Homegeneous Response Groups, which I explained in details in a post I wrote a little while ago.

PS : As @Placidia mentioned, the case of the 1936 election is really different : the bias comes from a flaw in the sampling frame, it is not a form of non-response biais.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.