Which test to use: Chi-squared, Fisher's exact or some other?

Question

I asked 200 survey participants the same sequence of eight multiple-choice questions with four answer options (A, B, C and D). Here are the results:

• All A's: 58
• All B's: 1
• All C's: 2
• All D's: 0
• Mixtures of A's, B's, C's and D's: 139

I want to work out whether these results are statistically significant, which I take to mean whether the probability that they occurred randomly is less than 0.001. I understand the expected value for each combination of answers – all A's, all B's, all C's, all D's, and each mixture of A's, B's, C's and D's – to be 200/(4^8), which is 0.003051758.

So here's the problem. I've read that a Chi-squared test requires all of the expected values to be greater than five, and in this case none of them is greater than five. I've also read that none of the observed values can be zero, and in this case one of them is zero. I've seen something about artificially combining categories to bring all the expected and observed values above five and zero respectively, but I don't understand how this can be done without artificially affecting the p-value. Finally, I've read a few things about Fisher's exact test, but all of them seem to suggest that I'd be allowed only a few rows of values, whereas in this case I have 65,536 (i.e. 4^8).

What is the most appropriate method in this circumstance?

Answer 1

With your concern about using Fisher's exact test, as I understand it the test can be applied to tables larger than 2x2, but the reason for this norm is that it is computationally expensive otherwise.

The greater concern related to Fisher's exact test would be the assumption of fixed totals. An example of this is given here. Quoted:

An example [of fixed totals] would be putting 12 female hermit crabs and 9 male hermit crabs in an aquarium with 7 red snail shells and 14 blue snail shells, then counting how many crabs of each sex chose each color (you know that each hermit crab will pick one shell to live in)

Your study design does not meet this condition.

Other options are:

• A G-test (likelihood ratio $\chi^2$). Some recommend using this when the sample is low. In R you can use the likelihood.test function in the Deducer package
• Exact multinomial test. Recommended here and elsewhere. The function 'xmulti' of the R package XNomial. Read the vignette.

Now that you've reached the limits of excel, give another software package a go.