2
$\begingroup$

While I was preparing an internet survey, I read from multiple sources which says it is more important to have a high response rate than large sample size.

Q1. I don't quite understand the logic behind this. Can you explain? If i can afford to survey the entire population, would that be better than a random sub-sample?

Also, the importance of random sampling is often stressed.

Q2. But what if I just invite the entire population of interest to the survey (because I can... using the internet). Wouldn't this be better than random sampling (even though the response rate might be lower because N is larger?)

I understand the answer to the questions above might depends on the objective of the study. If this is the case please delineate this for me. Thank you!

Update

After seeing seeing several responses, I guess I have asked a misleading question. I am not comparing response rate and sample size as in a contest of which is more important. I am trying to find out, if it costs me nothing more, relative to random sub-sampling, to survey the entire population of interest, is there any reason why I shouldn't do that and stick with a random sub-sample instead?

$\endgroup$
2
$\begingroup$

You are assuming that non-respondent data are missing at random (MAR). MAR is a very strong assumption, and not generally borne out in practice. You already know that respondents and non-respondents differ on at least one crucial dimension: non-respondents sent your instrument/link/request (whatever you might have used) to the bit-bucket. Respondents did not ignore your request.

If (and only if) you can show that data are MAR, then drawing a large sample and accepting the returns as a random sample makes sense. The problem is that you cannot show this without investigating the non-respondents, that is, by doing follow-up work.

In fact, the situation is worse than Frank suggests above: large and non-random samples are more likely to have large biases than smaller, random samples. Self-selected samples are rarely, if ever random.

I'll give you exactly the same advice I give researchers at my university when they propose ridiculously large samples without follow-up: don't do it. Take a realistic sample that is random on the appropriate levels. Plan to do follow-up, and be ready to be as big a pest as you must be to get the data. Don't be above bribery (that is, offering incentives to respondents) to get the data. Shoot for a response rate in the 90% range (it can be done, if you have the resources).

It's getting a little long in the tooth, but Lessler and Kalsbeek, Nonsampling error in surveys is still a good go-to reference on these issues.

Responding to your update: Yes, it matters. It matters for follow-up purposes. Suppose that you can draw a random sample of $n=100$ or a census of $N=10,000$, with a deterministic response rate of 80% in either case. In the case of random sampling, the non-respondent group has 20 members. In the case of the census the non-respondent group has 2,000 members. If you have no intention of doing any follow-up no statistician I know can help you.

If you do intend to do follow-up (and you should), then in the case of the sample a census of the non-respondents is feasible. You can be as big a pest as necessary to get those 20 observations. Follow-up of 2,000 is a different issue entirely. Do you propose to draw a sample of those to reduce the number to something manageable? If so, why didn't you just sample in the first place?

$\endgroup$
4
  • $\begingroup$ my 'follow up' question to you. ^_^ 1. I still don't understand how missing mechanism relates to my question. Say it is MNAR. Say the population can be split into 90/10 according to the variable of interest X with 2 levels (Yes/No), and say those with X=No never response while the other always will. If I survey the entire population, I will get 90% response rate with Pr(X=yes) estimated to be 100%. If I survey a random sample of size n, I will still get a response rate of 90% and the same estimate of 100%. What's the difference? $\endgroup$
    – qoheleth
    Jul 8 '14 at 3:30
  • 1
    $\begingroup$ You've assumed that X="No" never respond. In that case, those individuals are effectively absent from your sampling frame. That is a non-coverage error in your frame. That problem is addressed by learning (1) that such individuals exist, and (2) how to move them into the respondent category. If you have a frame that includes both X="Yes" and X="No " you can (begin to) learn these things with appropriate follow-up. In a sample of size $n=100$ the expected number of "Nos" is 10, the probability of 0 "No" in the sample is $2\times 10^{-5}$. Followup of ~10 non-respondents isn't onerous. $\endgroup$
    – Dennis
    Jul 8 '14 at 3:55
  • 2
    $\begingroup$ Further to my comment, your expected 90 respondents do not make a random sample of the population. The random sample are the 100 you selected. Ten (10) are missing due to a systematic bias in your methods. $\endgroup$
    – Dennis
    Jul 8 '14 at 3:59
  • $\begingroup$ Both of your comments seem to reinforce my question though, but you comments seem to indicate that, whether I survey the entire population or a sub-sample, which I randomly select, the conclusion I can draw from those who respond will still be subject to the same bias. And if that's the case, what's stopping me from surveying the entire population? However, 1 line does seem to be relevant ' Followup of ~10 non-respondents isn't onerous.' So perhaps the only reason why I should choose a sub-sample is because it is easier to do followup (which increases final response rate) $\endgroup$
    – qoheleth
    Jul 9 '14 at 0:07
2
$\begingroup$

The problem is that bias is not overcome by a large sample size. A survey of 100 randomly chosen subjects with a 1.0 response proportion would yield much more valuable information that a 0.10 response fraction from 10,000 subjects surveyed, yielding 1000 subjects responding. Suppose for example that one were estimating the probability that opinion X is held and the true probability is 0.05. A 0.95 response fraction would yield an estimate of 0.0 if only persons not holding opinion X responded. The estimate would be off by huge relative amount.

$\endgroup$
4
  • $\begingroup$ I guess that's my question. If your response rate is 0.1 for 10,000, how can you guarantee that the response rate for the 100 random samples be any higher? If the response rate for the 100 is also 0.1, then would it be better to have surveyed 10,000? $\endgroup$
    – qoheleth
    Jul 8 '14 at 2:40
  • 1
    $\begingroup$ No. It appears to me that you are conflating "random sample" with "responses". The two are identical only if the response rate is 100%, $\endgroup$
    – Dennis
    Jul 8 '14 at 4:14
  • 1
    $\begingroup$ You can raise the response rate by targeting fewer subjects if you (1) provide incentives such as cash payments, (2) reduce the response burden by randomizing which section of a long survey each subject is given, or (3) you repeatedly follow-up the smaller number of subjects to urge non-respondents to respond. $\endgroup$ Jul 8 '14 at 12:05
  • $\begingroup$ So it seems that the reason I should only survey a sub-sample instead of the entire population is because it will be easier to do follow-up, which will ultimately increase my response rate. And this reason really has nothing to do with the sampling scheme being random or not. $\endgroup$
    – qoheleth
    Jul 9 '14 at 0:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.