# How would I determine whether the differences I see in accuracy are actually significant?

I have a machine learning based NLP system. I run this system on set of sentences I obtained from different groups and I then measured my system's within group performance.

Table summarizes these results:

    GROUP_NAME   TruePred FalsePred   Accuracy
GROUP1        87        32         73.11
GROUP2         8         5         61.54
GROUP3        27         3         90.00
GROUP4         9         0        100.00
GROUP5        19         5         79.17


Well, system achieves 79% accuracy on sentences obtained from GROUP5 but for example it only gets 61% accuracy for group2.

I want to know if those pairwise differences are statistically significant or not. What am I supposed to do to determine this ?

Thanks in advance.

• if those are predictions on a test set, the number of good predictions is binomially distributed Oct 28 '19 at 17:48
• @carlo yes they are predictions on test set. So, what does having a binomially distributed predictions tell me? Could you give me some more hints ? Oct 28 '19 at 17:54
• then you can compare them using eg. Bernard's test: en.wikipedia.org/wiki/Barnard%27s_test. However, if you want to compare pairwise each group vs each other group be aware of multiple comparisons problem: en.wikipedia.org/wiki/Multiple_comparisons_problem Nov 18 '19 at 18:29

## 1 Answer

If you are worried about controlling your familywise Type I error rate (FWER), you can do an omnibus test to check for any differences before doing all the pairwise comparisons.

# omnibus test
data <- matrix(c(87, 8, 27, 9, 19, 32, 5, 3, 0, 5), ncol = 2,
dimnames =
list(c("Group1", "Group2", "Group3", "Group4", "Group5"),
c("TruePred", "FalsePred")))
data
fisher.test(data)
chisq.test(data)


The small cell counts will make many readers look for Fisher's exact test, but the Chi-square test gives a very close result. If you found a significant difference here, you could then explore pairwise comparisons.

If you wanted to jump straight into pairwise comparisons, you could use a more stringent significance threshold to control the FWER, e.g. 0.05/10, or say forget FWER and use 0.05 for all comparisons. Hat Tip to Wassermann for suggesting Barnard's test for the pairwise comparisons.

## Group 2 v 4

data24 <- matrix(c(9, 8, 0, 5), ncol = 2,
dimnames =
list(c("Group2", "Group4"),
c("TruePred", "FalsePred")))
data24
fisher.test(data24)
install.packages("Barnard")
Barnard::barnard.test(9, 8, 0, 5)


What approach you should take will depend on your goals and your audience. If you are genuinely interested in the group effects, I would push for collecting more data if at all possible. Any evidence generated by these small sample sizes will be suggestive at best. For example, I'm not big on worrying about FWER, but I remain pretty skeptical of the Group 2 v 4 difference.