How would I determine whether the differences I see in accuracy are actually significant? I have a machine learning based NLP system. I run this system on set of sentences  I obtained from different groups and I then measured my system's within group performance.
Table summarizes these results:
    GROUP_NAME   TruePred FalsePred   Accuracy
    GROUP1        87        32         73.11
    GROUP2         8         5         61.54
    GROUP3        27         3         90.00
    GROUP4         9         0        100.00
    GROUP5        19         5         79.17

Well, system achieves 79% accuracy on sentences obtained from GROUP5 but for example it only gets 61% accuracy for group2. 
I want to know if those pairwise differences are statistically significant or not. What am I supposed to do to determine this ?
Thanks in advance.
 A: If you are worried about controlling your familywise Type I error rate (FWER), you can do an omnibus test to check for any differences before doing all the pairwise comparisons. 
# omnibus test
data <- matrix(c(87, 8, 27, 9, 19, 32, 5, 3, 0, 5), ncol = 2,
               dimnames =
                 list(c("Group1", "Group2", "Group3", "Group4", "Group5"),
                      c("TruePred", "FalsePred")))
data
fisher.test(data)
chisq.test(data)

The small cell counts will make many readers look for Fisher's exact test, but the Chi-square test gives a very close result. If you found a significant difference here, you could then explore pairwise comparisons.
If you wanted to jump straight into pairwise comparisons, you could use a more stringent significance threshold to control the FWER, e.g. 0.05/10, or say forget FWER and use 0.05 for all comparisons. Hat Tip to Wassermann for suggesting Barnard's test for the pairwise comparisons.
## Group 2 v 4

data24 <- matrix(c(9, 8, 0, 5), ncol = 2,
                 dimnames =
                   list(c("Group2", "Group4"),
                        c("TruePred", "FalsePred")))
data24
fisher.test(data24)
install.packages("Barnard")
Barnard::barnard.test(9, 8, 0, 5)

What approach you should take will depend on your goals and your audience. If you are genuinely interested in the group effects, I would push for collecting more data if at all possible. Any evidence generated by these small sample sizes will be suggestive at best. For example, I'm not big on worrying about FWER, but I remain pretty skeptical of the Group 2 v 4 difference.  
