# Which test to use: Chi-squared, Fisher's exact or some other?

I asked 200 survey participants the same sequence of eight multiple-choice questions with four answer options (A, B, C and D). Here are the results:

• All A's: 58
• All B's: 1
• All C's: 2
• All D's: 0
• Mixtures of A's, B's, C's and D's: 139

I want to work out whether these results are statistically significant, which I take to mean whether the probability that they occurred randomly is less than 0.001. I understand the expected value for each combination of answers – all A's, all B's, all C's, all D's, and each mixture of A's, B's, C's and D's – to be 200/(4^8), which is 0.003051758.

So here's the problem. I've read that a Chi-squared test requires all of the expected values to be greater than five, and in this case none of them is greater than five. I've also read that none of the observed values can be zero, and in this case one of them is zero. I've seen something about artificially combining categories to bring all the expected and observed values above five and zero respectively, but I don't understand how this can be done without artificially affecting the p-value. Finally, I've read a few things about Fisher's exact test, but all of them seem to suggest that I'd be allowed only a few rows of values, whereas in this case I have 65,536 (i.e. 4^8).

What is the most appropriate method in this circumstance?

• By your comment on Fisher's exact test are you referring to the 'norm' of limiting it to 2x2 tables? Otherwise, please provide a link to the reference in question. Jan 2 '17 at 12:01
• You've caught me out there. What I'm alluding to is the inability of any free online calculators or Excel functions I've found to handle more than six rows. Suffice it to say I haven't the mathematical ability to do the calculations manually. Jan 2 '17 at 13:00

With your concern about using Fisher's exact test, as I understand it the test can be applied to tables larger than 2x2, but the reason for this norm is that it is computationally expensive otherwise.

The greater concern related to Fisher's exact test would be the assumption of fixed totals. An example of this is given here. Quoted:

An example [of fixed totals] would be putting 12 female hermit crabs and 9 male hermit crabs in an aquarium with 7 red snail shells and 14 blue snail shells, then counting how many crabs of each sex chose each color (you know that each hermit crab will pick one shell to live in)

Your study design does not meet this condition.

Other options are:

• A G-test (likelihood ratio $\chi^2$). Some recommend using this when the sample is low. In R you can use the likelihood.test function in the Deducer package
• Exact multinomial test. Recommended here and elsewhere. The function 'xmulti' of the R package XNomial. Read the vignette.

Now that you've reached the limits of excel, give another software package a go.

• Conditioning on marginal totals has always been controversial. I don't really see it as an issue. Because the exact test has to look at calculating multinomial probabilities for all table as extreme or more extreme than the given table it can be computer intensive even by today's standards. Jan 2 '17 at 14:04
• I looked at the reference you (Simon) link about the chi square test. It contains a lot of good information and references but I did not see anything about the asymptotic nature of it. Jan 2 '17 at 14:11
• @Michael, I see that I have my own homework to do digesting this among other material. Jan 2 '17 at 14:32
• Putting aside any judgment about the links you provide I liked your answer and gave an upvote. If you want to invest time in studying the asymptotic properties of Pearson's chi square by all means go for it. Jan 2 '17 at 14:50
• Thanks for your replies. I'll have a look at those links. In the meantime, can you please confirm whether my calculation of expected values is correct? Jan 2 '17 at 16:23