# Statistical significance of a difference between poll percentages with a small sample?

Analyzing my company’s internal survey results, I was stuck with calculating statistical difference between percentages. I usually used a Z-test as described below, but I am not sure whether it is suitable for very small samples (5 to 10 respondents), and if it is not, then how to calculate the difference?

Now let me describe my common approach. Suppose there is a question with multiple answer options (let them be A, B, etc.), and each respondent can choose only one option. Then, in my case, we have n = 259 participants with A = 117 of them choosing A answer, and B = 88 choosing B answer. (The other 54 chose other options or were undecided.)

First, I calculate the percentages: $a = {A \over n} \approx 0.45$, $b = {B \over n} \approx 0.34$, and the difference $\Delta = {{\left| {A - B} \right|} \over n} \approx 0.11$.

Second, I calculate the standard error of difference, as suggested by Wikipedia (based on the book Survey Sampling by Leslie Kish, p. 500):

$$\bar \sigma = \sqrt {{{a + b - {{{\rm{(}}a - b{\rm{)}}}^2}} \over n}} = 0.05484\ldots$$

And finally, I calculate $Z = {\Delta \over {\bar \sigma }} \approx 2.04$ and compare it to the critical Z-score = 1.96 (for 95% level of confidence, two-tailed test). Since my Z is greater than the critical one, I reject the hypothesis that there is no difference between percentages.

But now a question arises: does this approach work for small samples (say, 7 or 9 respondents)? If it doesn’t, then how to calculate statistical difference in this case? If you will find it interesting to answer, then I would ask you please to give a calculations’ example, for I am not a skilled statistician, I perform all the math in MS Excel, and I will hardly understand the procedure without a live example.

• The calculation of 'a' and 'b' are incorrect, as they should sum to 1 for this test to work (i.e. n is really the number of responses you are considering in the test, 117 + 88).
– khol
Apr 22, 2018 at 18:57
• Additionally, it is likely incorrect, depending on the exact question you are asking, to group together / ignore other responses.
– khol
Apr 22, 2018 at 18:58
• @khol, did I understand you properly, that the examples in Wikipedia and in Kish’s book are incorrect?
– user185547
Apr 22, 2018 at 19:13
• No. The test from wiki looks correct (when dealing with sufficiently large samples), but it was incorrectly applied above. To run the test correctly, the proportions should be relative to the set of responses we are considering (n = 88 + 117 = 205), but instead 'a' and 'b' are relative to the entire set of responses (n = 259) .
– khol
Apr 22, 2018 at 19:39
• I am sorry, @khol, but Kish derives the formula for variance of proportions’ difference with the special assumption of more than two proportions. See the excerpt from the book, p. 501.
– user185547
Apr 22, 2018 at 19:57