I have a machine learning based NLP system. I run this system on set of sentences I obtained from different groups and I then measured my system's within group performance.

Table summarizes these results:

    GROUP_NAME   TruePred FalsePred   Accuracy
    GROUP1        87        32         73.11
    GROUP2         8         5         61.54
    GROUP3        27         3         90.00
    GROUP4         9         0        100.00
    GROUP5        19         5         79.17

Well, system achieves 79% accuracy on sentences obtained from GROUP5 but for example it only gets 61% accuracy for group2.

I want to know if those pairwise differences are statistically significant or not. What am I supposed to do to determine this ?

Thanks in advance.

  • $\begingroup$ if those are predictions on a test set, the number of good predictions is binomially distributed $\endgroup$
    – carlo
    Oct 28, 2019 at 17:48
  • $\begingroup$ @carlo yes they are predictions on test set. So, what does having a binomially distributed predictions tell me? Could you give me some more hints ? $\endgroup$
    – zwlayer
    Oct 28, 2019 at 17:54
  • 1
    $\begingroup$ then you can compare them using eg. Bernard's test: en.wikipedia.org/wiki/Barnard%27s_test. However, if you want to compare pairwise each group vs each other group be aware of multiple comparisons problem: en.wikipedia.org/wiki/Multiple_comparisons_problem $\endgroup$
    – Wassermann
    Nov 18, 2019 at 18:29

1 Answer 1


If you are worried about controlling your familywise Type I error rate (FWER), you can do an omnibus test to check for any differences before doing all the pairwise comparisons.

# omnibus test
data <- matrix(c(87, 8, 27, 9, 19, 32, 5, 3, 0, 5), ncol = 2,
               dimnames =
                 list(c("Group1", "Group2", "Group3", "Group4", "Group5"),
                      c("TruePred", "FalsePred")))

The small cell counts will make many readers look for Fisher's exact test, but the Chi-square test gives a very close result. If you found a significant difference here, you could then explore pairwise comparisons.

If you wanted to jump straight into pairwise comparisons, you could use a more stringent significance threshold to control the FWER, e.g. 0.05/10, or say forget FWER and use 0.05 for all comparisons. Hat Tip to Wassermann for suggesting Barnard's test for the pairwise comparisons.

## Group 2 v 4

data24 <- matrix(c(9, 8, 0, 5), ncol = 2,
                 dimnames =
                   list(c("Group2", "Group4"),
                        c("TruePred", "FalsePred")))
Barnard::barnard.test(9, 8, 0, 5)

What approach you should take will depend on your goals and your audience. If you are genuinely interested in the group effects, I would push for collecting more data if at all possible. Any evidence generated by these small sample sizes will be suggestive at best. For example, I'm not big on worrying about FWER, but I remain pretty skeptical of the Group 2 v 4 difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.