Should I use Fisher's exact test for this?

Question

I'm relatively new to statistics and I'm a little confused about which test I should be using for this problem. Let’s say I am collecting data about responses to a book under two different conditions (C1 and C2). In each condition, I am looking at several attributes about the book and asking for ratings using a 5 point Likert scale.

The example tables below show the responses for the “enjoyability” attribute of the book, out of sample size of, say, 22.

C1
Rating:        1   2   3   4   5
Frequency:     2   1   2   9   8

C2
Rating:        1   2   3   4   5
Frequency:     10  6   3   3   0

I would like to compare the “enjoyability” ratings under each condition and test for significance (is the book significantly more enjoyable under C1?). Is Fisher’s exact test what I should be using? Is there another test which might be more suitable?

Answer 1

You're testing for a location difference on ordered categorical response. I don't see how Fisher's exact test really relates to it*.

*(but see my final note relating to exact tests more generally -- if you come up with a suitable measure of the shift in the distribution that respects the known ordering, an exact test could be constructed by enumeration in small tables or by simulation in larger ones)

Do you have the same people under both conditions or different people?

If its the same people, the analysis will be quite different.

I thought that Fisher's exact test could be used for categorical data

Well, yes it can be used for categorical data, but that's almost beside the point.

It's not suited for answering the question you want to ask. If you keep the pairing in mind you could construct a contingency table but it doesn't let you discern directional preferences.

I am assuming category 5 is the most enjoyable and 1 is the least. If I have that backward you may have to 'flip' parts of my discussion about.

A suitable analysis is to look - for each individual - at their condition 1 and condition 2 scores and see how they compared.

You have a couple of options -

Option 1: treat the Likert scale as only ordinal. You can for each participant score as follows:

 1: Preferred condition 2 to condition 1
 0: Rated both conditions in the same category
-1: Preferred condition 1 to condition 2

You could do a sign test (basically, throw out the zeroes).

Option 2: treat the Likert scale as interval (which is very common - every time people add Likert scale items to produce an overall score, they're doing this). That is, treat the category scores as actual numbers.

You could then either consider something like a Wilcoxon signed rank test (but taking account of the fact that there are a lot of ties), or perhaps a paired-t-test.

There are other possibilities; these would be relatively 'standard' ones.

---

Edited to add:

You may be able to construct an exact test - a permutation test would be an exact test in the intended sense, for example - but doing this would involve using some statistic to order the samples in a way that doesn't apply to the test conceived by Fisher (who ordered his tables by probability, which works fine for a two-way test on a 2x2 table but which is difficult to generalize)

Answer 2

You need a repeated measures test for ordinal data - the Wilcoxon signed ranks test is probably the one to go for. http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test

You don't have enough information in these tables, because you don't have information about the person. You need to be able to link the two scores from one person together, so:

ID S1 S2
A  3   4
B  2   5
etc.

Here's an explanation in R: http://www.r-tutor.com/elementary-statistics/non-parametric-methods/wilcoxon-signed-rank-test