Suppose I have a study of 3031 people that obtains responses to various questions (95% CL). One of the questions (Q1a) gets a yes response from 616 of the people. Of those 616, only 34 have a QualityB (the remaining 582 don't) that is dependent on a yes answer to (Q1a).

When determining the margin of error for QualityB (Confidence Interval) using this online calculator, do I use 34 or 616 for the sample size? I think I should use 616 for the sample size, 3031 for population size, and 5% for percentage, but I am not sure.

  • Using 616 as the sample size, result is +/- 1.54%.
  • Using 34 as the sample size, result is +/- 7.29%.
  • $\begingroup$ You should not have used 3031 as the population size in the online calculator! Population size is probably some millions of people. Using 34 as the sample size does not make any sense at all, because this would imply that you only surveyed 34 people. So what I think remains as a reasonable question, is whether to use 616 or 3031 as the sample size. The answer given by whuber (+1) is that you should use 616. The caveat pointed out by Michael is that you can only estimate the prevalence of QualityB among yes-answers, not among the whole population. $\endgroup$ – amoeba Jan 8 '16 at 16:36

As I interpret the question, this problem can easily be resolved through some careful reasoning about what is going on and what the survey objectives are. We can even do some simple mental calculations as a check on the confidence interval produced by the software.

Consider this model of the survey. In a population, members have the following attributes: their answer to Q1a ("yes" or "no") and their QualityB (present or absent). However, a QualityB value is available and meaningful only for those who answered "yes". One purpose of the survey is to estimate the proportion of "yes" answerers who have QualityB present. To this end, the survey has selected 3031 people independently and randomly from the population.

If this model reasonably approximates the survey and its objective, then notice that the randomization procedure by which all 3031 people were selected constitutes a fortiori a random procedure to select among the "yes" answerers. However (unlike specifying the sample size of 3031, which is usually determined by the investigator), the number of "yes" answerers was not determined in advance: it, too, is a random quantity.

Nevertheless, in part because the subsample size of 616 is so large, it is a reasonable approximation to analyze it as if it were a random sample of 616 people chosen from just the "yes" answerers in the population. (As a partial justification of why we can consider 616 "large," by using the approximate Binomial theory calculations below, one can figure out that a comparable survey would likely have included between 579 and 653 "yes" answerers; this amount of variation would not change the calculation of the confidence limits much at all.) Accordingly, the analysis of the 34 people with QualityB present can proceed as usual.

The binomial theory is applicable: we estimate the proportion of QualityB people out of the "yes" answerers as $34/616$ = $5.5$% and we estimate the variance of that proportion as $(34/616)(1 - 34/616)$, for an estimated standard deviation of $0.22836$. Because the subsample size is $616$, the standard error of the proportion is $0.22836/\sqrt{616}$ = $0.92$%. So--just to see where this is headed--we could use a normal approximation as a rough check. This tells us to expect the confidence interval procedure to give us a range from about $5.5$ - $1.65 \times 0.92$ = $4.0$% to $5.5 + 1.65 \times 0.92$ = $7.0$%. This range is very close to the quoted value of $5.5 \pm 1.54$%. (The multiplier of $1.65$ should give, approximately, a $90$% two-sided confidence interval.)

We conclude that the proportion of people with QualityB among all "yes" answerers in the population is likely to be between $4$% and $7$%. To deduce this, we have used a procedure that will mislead us (by the luck of the draw) at most $10$% of the time it is appropriately applied; that's where our "confidence" comes from.


Because some questions about the validity of this answer have been raised in comments, let's check. One way is to bootstrap the data to assess the bias. But before proceeding, let's recast the problem in a more concrete form.

Suppose, then, we are interested in the proportion of U.S. senior citizens (defined, say, as age 55 or older on January 1, 2012 and were resident in the U.S. on that date) who have ever tried recreational drugs. To this end, we identify all resident adults and send out a questionnaire to 3011 randomly selected adults. On it are two questions, analogs of the Q1a and QualityB questions discussed earlier:

  1. What was your age on January 1, 2012?

  2. If you answered 55 or older to question 1, have you ever knowingly consumed a drug, for recreational purposes, that at the time either required a physician's prescription or was illegal to use or sell in the U.S.?

Miraculously--perhaps through incredibly diligent followup--you receive valid responses on all 3011 questionnaires. The data are:

  • 616 of the responses are ages 55 or older.

  • Of those 616, 34 answered "yes" to the second question.

What proportion should you estimate? Is there any valid way to estimate a proportion at all?

One form of the bootstrap studies this problem by adopting a synthetic population having exactly the same proportions observed in the data and recreates the experiment and its analysis many, many times, independently. Here is reproducible R code to do that for 100,000 independent trials, using the Binomial estimate recommended above:

trial <- function(n.trials, n=1, p1=1/2, p2=1/2) {
  x <- rmultinom(n.trials, n, c(p2,1-p2) %o% c(p1,1-p1))
  m <- x[1,]+x[2,] # Total who answer the second question
  mean <- x[1,]/m  # Proportion of "yeses" in the second question
  se <- sqrt(mean*(1-mean)/m)
  rbind(mean, se)  # Estimate and standard error of the estimate for each trial
sim <- trial(100000, 3031, 616/3031, 34/616)

The average estimate, mean(sim[1,), is $0.0551537$: almost identical to the correct value of $34/616 \approx 0.0551948$ in the synthetic population. There is no bias.

How about the approximate confidence interval procedure? We can check in each of the 100,000 trials whether the confidence interval covered the true value of $34/616$ or not:

coverage.upper <- sim[2,] * 1.65 + sim[1,] > 34/616
coverage.lower <- -sim[2,] * 1.65 + sim[1,] < 34/616
(sum(coverage.upper) + sum(coverage.lower))/100000 - 1

The result, $0.89542$, is within one-half of one percent of the desired coverage of $0.90$: that's excellent, especially given the approximations that were made.

Given these (hypothetical) data we may legitimately conclude, then, that approximately $5.52$% of all U.S. senior citizens have used recreational drugs. With $90$% confidence that proportion is between $4$% and $7$%.

| cite | improve this answer | |
  • $\begingroup$ I don't see why the fact that the 616 is a relatively large sample allows you to consider it to be random. $\endgroup$ – Michael R. Chernick Sep 1 '12 at 1:10
  • 1
    $\begingroup$ The 616 that result in yes answers is random but that does not make selecting the number from those 616 that answered with quality B a sample that is like a random sample from the original population. Suppose we were measuring hip to waist ratio from a population that consisted of 50% men and 50% women. We ask a random sample of 1000 people from this population whether or not they wear lipstick. the 497 women answer yes and the 503 men in the sample answer no. We then sample 100 from the 497 women that answered yes and measure the hip to waist ratio. $\endgroup$ – Michael R. Chernick Sep 1 '12 at 16:43
  • 2
    $\begingroup$ You seem to have drifted away from the current context, @Michael. Yes, what you describe is silly, but I see no valid parallel between it and what I am discussing here. $\endgroup$ – whuber Sep 1 '12 at 16:54
  • 3
    $\begingroup$ This question has a different objective, @Michael. It sounds like you have not really read my answer, because you seem to have adopted a different model of what is going on. $\endgroup$ – whuber Sep 1 '12 at 17:00
  • 1
    $\begingroup$ OK, @Michael, to reduce the chance that my exposition might be misunderstood, I added a section with further justification. $\endgroup$ – whuber Sep 1 '12 at 21:11

No the 616 does not constitute a random sample from the population. It is a specific subset that answered yes to question 1a. Is quality B only assessed on those 616 or is it also studied for the remaining 2415? There is no simple answer to your question. You could bootstrap the 3031 and for each bootstrap sample you will get a subset of the 616 that answered yes to Q1a and from that you will have a subset of 34 with quality B. The bootstrap will give you a distribution of percentages from which you could construct a bootstrap confidence interval. Sorry to make it complicated but I don't see an easy way.

| cite | improve this answer | |
  • 1
    $\begingroup$ Clearly our two answers are contradictory, but that does not mean one is necessarily wrong: it likely indicates a difference of interpretation of the question. Unfortunately, that difference is difficult to identify in the absence of any analysis on your part. I have done what I could to make the context and applicability of my answer as clear as possible. Could you provide anything to support your answer, Michael, besides a personal opinion and a "silly" example (I am quoting you, as you know)? A theoretical analysis? A simulation? A reference? $\endgroup$ – whuber Sep 3 '12 at 14:29
  • 3
    $\begingroup$ Maybe you should study (and reproduce) the simulation, Michael: it is there because it clearly and unambiguously shows how I have interpreted the question. It also clearly shows that the answer I gave, for this interpretation, is correct. I am also able to provide a theoretical (read: mathematical) analysis of this interpretation to support the statements I made, but in light of the simulation results this seems like too much work for little additional gain. I submit, then, that the onus is on you to (a) clarify your interpretation and (b) demonstrate the correctness of your claims. $\endgroup$ – whuber Sep 3 '12 at 15:00
  • 1
    $\begingroup$ @whuber The OP seems only interested in what denominator to use to estimate the proportion of subjects with Quality B and how to generate the appropriate confidence interval for it. In your answer you are assuming that the OP wants to know if we look at the population of subjects that answer yes to Q1a, the proportion that would have quality B. I grant that if that is the correct interpretation your answer would be correct. But I don't see where that is explicitly or implicitly the correct interpetation. $\endgroup$ – Michael R. Chernick Sep 3 '12 at 15:19
  • 1
    $\begingroup$ My interrpetation is that he wants the proportion from the population of 3031 that have quality B regardless of the answer to Q1a. It just happens that the survey only provides an answet about quality B based on a yes answer to Q1a. In that case the potential bias in the 616 that answered yes matters and my answer would apply. This is a matter of interpetation of the question and does not require analysis or further jsutification for my answer. I am a little bothered that someone downvoted my answer presumably because it appears to conflict with yours. $\endgroup$ – Michael R. Chernick Sep 3 '12 at 15:24
  • 4
    $\begingroup$ And someone downvoted my answer, too--but that's beside the point. Thank you for explaining your interpretation: finally, it is possible to assess your answer on its merits. Re "require analysis": In general, the answers that are appreciated on Stack Exchange do offer analysis and justification in addition to opinions and recommendations. You can expect to receive downvotes for answers that do not have such support--and, as my experience in this thread shows, sometimes demonstrably correct answers with such support get (silent) downvotes too, however unfair that might be. $\endgroup$ – whuber Sep 3 '12 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.