# Analogue of fisher's exact test for one group?

I understand that Fisher's exact test applies to testing whether the proportion of an outcome in one group, $$p_1$$, differs from the proportion of the outcome in another group, $$p_2$$. This could be like drug efficacy in group 1 vs drug efficacy in group 2.

Is there an analogue that works to test $$p_1$$ vs $$p_2$$ for the same group? The setting I'm thinking of is you have two classifiers making predictions on the same group, and classifier 1 has accuracy $$p_1$$ and classifier 2 has accuracy $$p_2$$. Given that classifier 1 got $$k_1$$ out of $$n$$ samples right and classifier 2 got $$k_2$$ out of the same $$n$$ samples right, how do we test whether $$p_1 > p_2$$ analogously to Fisher's exact test?

• It sounds like you want McNemar's test.
– Noah
Commented Mar 22 at 22:47

Consider the samples where the two tests gave different answers, and write $$m$$ for the number of these and $$m_1$$ for the number where test 1 was postive and test 2 negative (and $$m_2=m-m_1$$ for the others). Under the null hypothesis $$p_1=p_2$$, $$m_1\sim Binomial(m,1/2)$$, giving an exact test.
This (as @noah says) is McNemar's test. The $$\chi^2$$ approximation to the test is that $$(m_1-m_2)^2/(m_1+m_2)\sim \chi^2_1$$ giving one of the few tests you can easily do in your head.