Statistical significance of a difference between poll percentages with a small sample? Analyzing my company’s internal survey results, I was stuck with calculating statistical difference between percentages. I usually used a Z-test as described below, but I am not sure whether it is suitable for very small samples (5 to 10 respondents), and if it is not, then how to calculate the difference?
Now let me describe my common approach. Suppose there is a question with multiple answer options (let them be A, B, etc.), and each respondent can choose only one option. Then, in my case, we have n = 259 participants with A = 117 of them choosing A answer, and B = 88 choosing B answer. (The other 54 chose other options or were undecided.)
First, I calculate the percentages: $a = {A \over n} \approx 0.45$, $b = {B \over n} \approx 0.34$, and the difference $\Delta  = {{\left| {A - B} \right|} \over n} \approx 0.11$.
Second, I calculate the standard error of difference, as suggested by Wikipedia (based on the book Survey Sampling by Leslie Kish, p. 500):
$$\bar \sigma  = \sqrt {{{a + b - {{{\rm{(}}a - b{\rm{)}}}^2}} \over n}} = 0.05484\ldots$$
And finally, I calculate $Z = {\Delta  \over {\bar \sigma }} \approx 2.04$ and compare it to the critical Z-score = 1.96 (for 95% level of confidence, two-tailed test). Since my Z is greater than the critical one, I reject the hypothesis that there is no difference between percentages.
But now a question arises: does this approach work for small samples (say, 7 or 9 respondents)? If it doesn’t, then how to calculate statistical difference in this case? If you will find it interesting to answer, then I would ask you please to give a calculations’ example, for I am not a skilled statistician, I perform all the math in MS Excel, and I will hardly understand the procedure without a live example.
 A: No, you can not use the z distribution when you have a small sample size like that.  The distribution of sample means approaches the normal distribution as the sample size gets larger.  With small samples, the distribution isn't very close to the normal distribution, so approximations using z-scores won't be very accurate.
The normal distribution is used to calculate probabilities across all possible samples.  A way to get around that is to calculate probabilities yourself, since you have a small number of possible samples.  This can be done via Fisher's test of significance and using the definition of probabilities.
There are a few examples already on websites like I post below.  I don't have time today to draft any unique examples, but can come back and post one later.
https://mathworld.wolfram.com/FishersExactTest.html#:~:text=Fisher's%20exact%20test%20is%20a,associations%20between%20two%20categorical%20variables.&text=column%20sums%20and-,.,these%20probabilities%20must%20be%201.
https://en.wikipedia.org/wiki/Fisher's_exact_test
