# How to test whether two means of categorisation generate significantly different results

If I have the following contingency table in which Novel.Cat is a new means of classifying some thing. Ref.Cat is the gold standard. How can I do some sort of test to see whether the two means of classification generate significantly different results? Is just a simple Fisher's test appropriate? Or is there some other, simple way?

          Novel.Cat
Ref.Cat  a  b  c  d
a 15 2  0  1
b 9  9  0  0
c 5  1  5  0
d 0  0  0  6


With a Fisher's test, all I am testing is the non-independence between the groups. However, what I want to specifically test is the concordance between Novel.cat categorisation and Ref.cat categorisation. So, the ideal (best concordance) contingency table would be:

          Novel.Cat
Ref.Cat  a   b   c   d
a 18  0   0   0
b 0   18  0   0
c 0   0   11  0
d 0   0   0   6


Fisher's however, would say that the following results are significant (i.e. Novel.cat and Ref.cat are non-independent)

          Novel.Cat
Ref.Cat  a   b   c   d
a 0   0   0   18
b 0   0  18   0
c 11  0   0   0
d 0   6   0   6


However, quite clearly, Novel.cat has misclassified everything if Ref.cat is to be taken as the gold standard.

• Fisher's exact test seems utterly appropriate to me. It yields $p=2\times 10^{-9}$, so I guess that any other reasonable test will also reject the null. Of course, you will have to decide whether statistical significance equals practical relevance here, or whether e.g. some misclassifications are more problematic than others. – Stephan Kolassa Jan 7 '13 at 22:21
• So, interpreting this, would mean that Novel.cat and Ref.cat are not independent: they are related in some way. However, this could just mean that all Novel.cat a gets classified as b and similarly for the others (they don't above but they could to give non-independence). These are still misclassifications and Fisher's doesn't detect this. – Nodnol Jan 8 '13 at 9:09
• Sorry, I think I don't really understand your previous comment. Could you clarify? The interpretation is convoluted (as always in null hypothesis significance testing): under the assumption that Novel.Cat produces the same expected classifications as Ref.Cat (the null hypothesis), it would be very unlikely to see classifications as different as you have here. – Stephan Kolassa Jan 8 '13 at 9:12
• I'll update question since it's easier with more space – Nodnol Jan 8 '13 at 9:16