I was using this online calculator to work out McNemar's Test on this data set:

enter image description here

I get the results:

The two-tailed P value equals 0.1859

Chi squared equals 1.750 with 1 degrees of freedom.

Now I understand how to calculate the Chi squared but I have no idea where this P value comes from.

Also how does the P value relate to whether there is a significant difference between classifiers?

If someone could explain this like a program in JavaScript or something it would be so much clearer. Every explanation I see throws jargon all over the place.


Asymptotically the McNemar test statistic follows a chi-squared distribution with 1 degree of freedom. So, if $x_{obs}$ is your observed McNemar test statistic, the $p$ value is

$p = \text{Pr}\left\{ \chi^2_1 > x_{obs}\right\}$

but perhaps this is all the jargon and what not that you were saying you were confused about. What the statement above is saying is that the $p$ value is a probability calcuated under the chi-squared $(\chi^2_1)$ distribution. You can think of a probability as being an area under a particular curve (think back to integrals in calculus).

The curve in question here is the chi-squared density enter image description here

The $p$ value is the blue shaded area under the curve I have plotted.

The curve is defined by: $f(x)=\dfrac{e^{-x/2}}{\sqrt{2x}\Gamma(1/2)}$

Now sure you can try to calculate this by hand to get p-value but most programming languages have built in functions to calculate are under density curves. In R you can do this:

> pchisq(1.75, df=1,lower.tail=FALSE)
[1] 0.1858767
| cite | improve this answer | |
  • 1
    $\begingroup$ Sorry, are you sure that it is a two-sided p-value? $\endgroup$ – Sergio May 17 '14 at 21:13
  • $\begingroup$ The p value noted in OP's question looks like one-sided because it matches the one-sided calculation i did in R. $\endgroup$ – bdeonovic May 17 '14 at 21:24
  • $\begingroup$ I'd say that it is one-sided, because the chi-squared support is $[0,+\infty]$, so pchisq(-1.75,1)=0. Am I wrong? $\endgroup$ – Sergio May 17 '14 at 21:29
  • 2
    $\begingroup$ I'd fell better if $p=2\text{Pr}(\chi_1^2>1.75)=0.1859$ ;-) $\endgroup$ – Sergio May 17 '14 at 22:37
  • 1
    $\begingroup$ Regardless of semantics (is the test two-sided because it has power against departures from the null hypothesis in either of two directions, or one-sided because one tail area of the test statistic's density gives its size?) the expression $p = 2\text{Pr}\left\{ \chi^2_1 > x_{obs}\right\}$ is simply wrong - out by a factor of 2 as @Sergio says. $\endgroup$ – Scortchi - Reinstate Monica May 18 '14 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.