3
$\begingroup$

I've inherited a survey from an old employee that consists of 27 questions. We have on the order of 500 respondents but not all of them answered every question as some questions are of the type 'if yes then please answer this'. The survey has an overall reported margin of error of +/- 4% at the 95% confidence interval and I understand that this was derived from a table based on the CI and the number of respondents.

I wanted to understand what the table was doing and I found this formula to calculate margin of error:

$$ C =z\sqrt{\frac{p (1-p)}{N}} $$

Where z is 1.96, N is the number of respondents and p is the response distribution (I think) (for example 45% of respondents say 'Yes').

My problem is that after reading about margin of error it seems incorrect to have calculated a margin for the entire survey. It seems like a margin of error would need to be calculated for every response (p) to every question.

So, my question: is it an acceptable practice to calculate an overall margin of error for a survey or should I, in a survey with 27 questions, calculate a margin of error for each question (given they have different respondent numbers) and then take the max as the margin of error for the whole survey? Or should I do something else entirely?

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ CI should not be .95, it should be a z score for a given level (e.g. 1.96). What you have labeled "Resp" is usually called p. $\endgroup$ – Peter Flom♦ Feb 20 '13 at 16:09
1
$\begingroup$

Technically, you are quite correct. The confidence interval depends on both N and p. Furthermore, the classical equation for the confidence interval of a binomial proportion isn't always all that good. There is quite a literature on this subject, a good starting point is this article by Agresti and Coull. `R' has implemented various of the suggested alternatives to the simple formula.

However, in practice, it is quite common to cite one CI for all the questions. Often the CI for the different questions do not vary that much. The question then becomes which CI to use as the "general" one. I would hope that they chose one calculated with a relatively small N.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

When you are reporting point estimates from individual questions in the survey, you should calculate and report a margin of error for each question. You report the overall survey margin of error only when you're describing the survey in general.

The overall margin of error for a survey is calculated using the total number of respondents (sample size) and p = 0.5 (p stands for proportion, actually). This is because under p = 0.5 (due to mathematical properties of the formula), you obtain the maximum margin of error. Therefore, the overall survey margin of error should be the largest margin of error you can get for any question in the survey. If you plug in 0.5 for p and the survey sample size for N in the formula you give (plus any applicable finite population correction), you should get a margin of error that you report. Using N=500 and p = 0.5, I get a margin of error of approximately 4.3

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.