0
$\begingroup$

I want to make a survey with several questions and conclude those results for the whole population. For example, I want to estimate the following questions for my 1 million population : "do you have a dog", "are you single", "is pizza your favorite food", etc... and I want to conclude that (for example) "40% of the whole population has a dog", "44% of the whole population is single", "pizza is the favorite food of 60% of the whole population"

I am looking at the sample size needed for surveys on this page, and it says the formula for estimating the sample size of a survey is:

$$Sample\;size = \frac{\frac{z^2 \;\times \;p(1-p)}{e^2}}{1 + (\frac{z^2 \; \times \; p(1-p)}{e^2N})}$$

where

  • $e$ is the margin of error
  • $N$ is the population size.
  • $p$ is the proportion

For the online calculation, one gives as input the confidence level, the whole population size, and the margin of error. For a confidence interval of $95\%$ and a margin of error of $3\%$, for the 1 million population, it gives a sample size of 1066. However, what is the proportion here? How can I interpret it since I did not give any value for input?

Edit: If I survey >1066 persons (among the whole population), can I make several conclusions about the population? Or is this sample only able to estimate ONE thing? Can I use this sample to estimate several proportions, or can I only estimate one proportion? Also, are proportions used only for Yes/No questions?

If from my sample of 1066, half of them (proportion of 0.5) have a dog, can I automatically say that the true proportion of the population is in the 95% confidence interval = $]0.5-0.03,0.05+0.03[=]0.47,0.53[$ ? Or do I need to some calculations for the confidence interval?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

You don't necessarily need more samples to make multiple inferences, but you may want to keep in mind that as you make more estimates, you're more likely to make a wrong estimate somewhere. Your sample size of 1066 will allow you to estimate proportions that will be within 3 percentage points of the true value 95% of the time (and they'll be even closer if the proportion estimate is different from 50%). You can make that inference for any numer of proportion estimates, and you'll always have your 95% CI being 3 percentage points wide. But even a 95% CI will miss the true value 5% of the time, so as you generate more and more proportion estimates, it becomes more and more likely that at least one of them is off. This is what's known as the family-wise error rate (FWER). For any individual comparison, the 95% CI will contain the true proportion 95 out of 100 times. If you want to estimate 100 proportions, though, it's quite likely that at least one of the CIs is off - you'd need to sample quite a bit more to control the FWER at the same rate.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Is there any formula that calculates the sample size while taking FWER into account? I.e, a formula that calculates a bigger sample size to correct that? $\endgroup$
    – Numbermind
    Apr 12, 2021 at 21:18
  • $\begingroup$ Is there any statistical book or article with this formula? I don't want to site a website for a formula. $\endgroup$
    – Numbermind
    Apr 13, 2021 at 8:13
  • $\begingroup$ Is there a way to calculate a "Corrected" sample size to control the fwer at the same rate? Increasing the sample size in 5% is enough? 15%? 30%? $\endgroup$
    – Numbermind
    Apr 14, 2021 at 14:11
  • $\begingroup$ (+1) A formula would be -- at best -- a very rough approximation, because it would depend on how you expect the responses to be correlated. If you assume the responses to be uncorrelated (a practical worst case), you will find the sample size is prohibitively large. These considerations usually lead to long conversations between client and statistician concerning what really needs to be accomplished, what the budget (in time, labor, and expenses) really is, and how to design an effective survey instrument. $\endgroup$
    – whuber ♦
    Apr 15, 2021 at 16:58
1
$\begingroup$

The "worst-case" for the width of a Wald-based CI around a proportion is when p=0.5 (the variance is p(1-p)/n; for a fixed n, p(1-p) is maximised when p=0.5 as you can prove using calculus or show by graphing), so if you want to be safe, this is a sensible choice. If you were confident that most people have a dog (at least 0.8), few people were single (no more than 0.3), and almost everyone loves pizza (at least 0.9), you could use p=0.3 for the sample size calculation and get a smaller value. However, if you were wrong and half of the respondents were single, your 95% CI around that proportion would be wider than you'd required (but still narrower than required for the dog and pizza variables as long as they were further in magnitude from 0.5 than 0.3).

As you're only talking about estimating the proportions, there's no effect on the sample size if you're interested in more than one (even many) variables. The answer could become more complicated if you were testing hypothesis, e.g. that men are more or less likely to have a dog or to like pizza than women. With multiple variables, you would want enough data to provide the specified margin of error, so you'd use the proportion closest in magnitude to 0.5 in that case. Again, by using p=0.5, you can't end up with a too small sample size. Note that this sample size is the number with data and so you'll want to allow for non-response and missing data. If the survey is anything other than a simple random sample, there will also be design effects that should be taken into account.

Note that if you're using the finite population correction factor, you're saying that you want to estimate the proportion only on that population. While this is sometimes what we want to do, other times, we have in mind a superpopulation from which the population was drawn (which leads to interference and not just description even if we could capture data without error on the entire extant population). The effect of the FPC is so low for non-small populations that it makes little difference in practice if you ignore it .

Improve this answer
$\endgroup$
8
  • $\begingroup$ I'm okay with having a big sample size. If p=0.5 is safer, then 0.5 it is. My main question is if I can conclude those things for the population. In my example, can I conclude that "40% of the whole population has a dog", "44% of the whole population is single", "pizza is the favorite food of 60% of the whole population? (Assuming I got those percentages in my sample) $\endgroup$
    – Numbermind
    Apr 12, 2021 at 19:38
  • $\begingroup$ Can I conclude several things with one single sample or do I need a sample for each question. That is confusing $\endgroup$
    – Numbermind
    Apr 12, 2021 at 19:39
  • $\begingroup$ You would have a point estimate of, say, 0.40 (40%) for dog ownership. The margin of error would be slightly less than 0.03 for n=1066, but not enough to matter, and the 95% CI would be 0.37 to 0.43. If you repeated your sample many times, 95% of these samples would produce 95% CIs that included the true population proportion. You can't make probabilistic statements about this particular interval, but you would feel "confident" that the true population percentage was between 37% and 43%. $\endgroup$
    – user215517
    Apr 13, 2021 at 4:41
  • $\begingroup$ You can estimate as many proportions as you want with that sample size. If you were using hypothesis tests to compare dog ownership between males and females and between younger and older and between urban and rural, etc. and then doing the same things for pizza, etc. your risk of false positives from these tests would accumulate with each additional test. You might not care if your study is exploratory, intending to identify further topics for study, or you might care if your study is more confirmatory or would inform policy. This would be a "it depends" question with no single answer. $\endgroup$
    – user215517
    Apr 13, 2021 at 4:45
  • $\begingroup$ Is there any statistical book or article with this formula? I don't want to site a website for a formula. $\endgroup$
    – Numbermind
    Apr 13, 2021 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.