7
$\begingroup$

If a questionnaire was administered to an entire population (group of interest) and the response rate was 68% can the questionnaire results be generalized to the population (100%, including the 32% missing)? Can confidence intervals be used for the data subsets even though this is more like a census questionnaire?

$\endgroup$
  • $\begingroup$ In my opinion, this is one of the problems with a census: if the time/resources of doing a census was poured into a well-run survey, you'd probably get better results. $\endgroup$ – Wayne Mar 10 '13 at 17:21
  • $\begingroup$ Is it possible to try to survey those who originally did not respond? That is, do you know who did and who did not respond and do you have permission and resources to reach out to them? If so, you might be able to determine how they differ (or not) from those who did respond, and get a better idea of how your missing data is missing (as Peter Flom explains). $\endgroup$ – Wayne Mar 10 '13 at 17:23
  • $\begingroup$ @Wayne I don't see how this is an argument for a survey instead. If (using Peter Flom's example) the data are MAR because of income effects, this will apply just as much to the survey as to the census; in both cases there are techniques to address it. If it is NMAR you have problems either way. $\endgroup$ – Peter Ellis Mar 10 '13 at 18:50
  • $\begingroup$ @PeterEllis: My argument is based on a fixed amount of resources to do the survey/census. If you focus on, say, an appropriate 10% of your population you'd in theory have roughly 10x the resources to attempt to gain a higher response rate. Perhaps I'm being naive. $\endgroup$ – Wayne Mar 10 '13 at 20:24
  • $\begingroup$ @Wayne - ok, I can buy that argument, but obviously it will be specific to particular situations, depending on what tools are available, what if any economies of scale they have, etc. $\endgroup$ – Peter Ellis Mar 10 '13 at 23:27
4
$\begingroup$

In general, no, you cannot do this.

More detail: The problem is that the 32% may be different from the 68%. If all you know is that they are missing then you have no way of saying in what way they may be different. But it's complex. With any missing data problem, even one with a census, one key thing is to determine (if you can) why there are missing data. The standard classification is:

Missing completely at random (MCAR)- that is, there is no particular reason why they are missing. Perhaps coffee spilled on their records. Or perhaps a random computer glitch did not send them the questionnaire.

Missing at random (MAR: The reasons for the missingness are captured by data that you do have. E.g., it is known that people at both ends of the income scale are less likely to answer the phone which would give missing data on a phone interview. But if you know the income of people who don't respond, then the data may be MAR.

Not missing at random (NMAR): Neither of the above.

If data is MCAR then your 68% results can be used "as is" as estimates of the population values.

If data are MAR then there are various approaches; probably the most popular now is multiple imputation.

If data are NMAR then there is no perfect solution, but I have seen some work showing that multiple imputation works reasonably well unless the data are "REALLY NMAR" (I saw a presentation by Joe Schafer, who is one of the real experts on this).

$\endgroup$
  • 1
    $\begingroup$ what are your thoughts on confidence intervals when the whole population is known...what population is the inference aimed at $\endgroup$ – user20650 Mar 10 '13 at 13:17
  • 1
    $\begingroup$ This is an area where there is no strong consensus. My own view is that these usually don't make much sense. But some people talk about "super-populations" or "possible populations" or similar. $\endgroup$ – Peter Flom Mar 10 '13 at 13:19
  • $\begingroup$ @user20650 it depends on what you are doing it for. For official statistics it rarely makes sense; if you want Australia's unemployment rate and you have a genuine census (we don't, in case you are wondering) I don't see any point in confidence intervals for it. If however you want to know the relationship between unemployment and years of education and you fit a model to genuine census data, it makes sense to infer to a super-population and have confidence intervals for your parameters. $\endgroup$ – Peter Ellis Mar 10 '13 at 18:47
  • $\begingroup$ okay, if I assume that the 32% of missing data is MCAR and generalize the 68% to the population, it wouldn't make sense to compute 95% confidence intervals for the medians for each of the data subsets (subsets refers to the data collected for each question in the census questionnaire). There wouldn't be anything to infer, right? Thanks :) $\endgroup$ – nadia Mar 20 '13 at 10:49
  • 1
    $\begingroup$ That is how I view it. $\endgroup$ – Peter Flom Mar 20 '13 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.