4
$\begingroup$

Analyzing my company’s internal survey results, I was stuck with calculating statistical difference between percentages. I usually used a Z-test as described below, but I am not sure whether it is suitable for very small samples (5 to 10 respondents), and if it is not, then how to calculate the difference?

Now let me describe my common approach. Suppose there is a question with multiple answer options (let them be A, B, etc.), and each respondent can choose only one option. Then, in my case, we have n = 259 participants with A = 117 of them choosing A answer, and B = 88 choosing B answer. (The other 54 chose other options or were undecided.)

First, I calculate the percentages: $a = {A \over n} \approx 0.45$, $b = {B \over n} \approx 0.34$, and the difference $\Delta = {{\left| {A - B} \right|} \over n} \approx 0.11$.

Second, I calculate the standard error of difference, as suggested by Wikipedia (based on the book Survey Sampling by Leslie Kish, p. 500):

$$\bar \sigma = \sqrt {{{a + b - {{{\rm{(}}a - b{\rm{)}}}^2}} \over n}} = 0.05484\ldots$$

And finally, I calculate $Z = {\Delta \over {\bar \sigma }} \approx 2.04$ and compare it to the critical Z-score = 1.96 (for 95% level of confidence, two-tailed test). Since my Z is greater than the critical one, I reject the hypothesis that there is no difference between percentages.

But now a question arises: does this approach work for small samples (say, 7 or 9 respondents)? If it doesn’t, then how to calculate statistical difference in this case? If you will find it interesting to answer, then I would ask you please to give a calculations’ example, for I am not a skilled statistician, I perform all the math in MS Excel, and I will hardly understand the procedure without a live example.

$\endgroup$
6
  • $\begingroup$ The calculation of 'a' and 'b' are incorrect, as they should sum to 1 for this test to work (i.e. n is really the number of responses you are considering in the test, 117 + 88). $\endgroup$
    – khol
    Apr 22, 2018 at 18:57
  • $\begingroup$ Additionally, it is likely incorrect, depending on the exact question you are asking, to group together / ignore other responses. $\endgroup$
    – khol
    Apr 22, 2018 at 18:58
  • $\begingroup$ @khol, did I understand you properly, that the examples in Wikipedia and in Kish’s book are incorrect? $\endgroup$
    – user185547
    Apr 22, 2018 at 19:13
  • $\begingroup$ No. The test from wiki looks correct (when dealing with sufficiently large samples), but it was incorrectly applied above. To run the test correctly, the proportions should be relative to the set of responses we are considering (n = 88 + 117 = 205), but instead 'a' and 'b' are relative to the entire set of responses (n = 259) . $\endgroup$
    – khol
    Apr 22, 2018 at 19:39
  • $\begingroup$ I am sorry, @khol, but Kish derives the formula for variance of proportions’ difference with the special assumption of more than two proportions. See the excerpt from the book, p. 501. $\endgroup$
    – user185547
    Apr 22, 2018 at 19:57

1 Answer 1

1
$\begingroup$

No, you can not use the z distribution when you have a small sample size like that. The distribution of sample means approaches the normal distribution as the sample size gets larger. With small samples, the distribution isn't very close to the normal distribution, so approximations using z-scores won't be very accurate.

The normal distribution is used to calculate probabilities across all possible samples. A way to get around that is to calculate probabilities yourself, since you have a small number of possible samples. This can be done via Fisher's test of significance and using the definition of probabilities.

There are a few examples already on websites like I post below. I don't have time today to draft any unique examples, but can come back and post one later.

https://mathworld.wolfram.com/FishersExactTest.html#:~:text=Fisher's%20exact%20test%20is%20a,associations%20between%20two%20categorical%20variables.&text=column%20sums%20and-,.,these%20probabilities%20must%20be%201.

https://en.wikipedia.org/wiki/Fisher's_exact_test

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.