I just want to make sure I'm doing things right!

I created 2 algorithms that classify my data into 2 groups. The first one gives me an accuracy of 76.66% (sensitivity=0.76, specificity=0.78). The second one: accuracy 81.33% (sensitivity=0.86, specificity=0.76)

So i wanted to know if this improvement is "statistically significant". I used McNemar's test (statistical test used on paired nominal data. It is applied to 2 × 2 contingency tables).

Is it a "good" way to proceed? if not which test should I choose?

  • $\begingroup$ Yes, a 2x2 contingency table would be appropriate to assess the null $H_0: \text{accuracy of both methods the same}$ $\endgroup$ – Gregg H Apr 25 '18 at 13:18
  • $\begingroup$ Thank you! What would be the null hypothesis? $\endgroup$ – learneRS Apr 27 '18 at 7:40
  • $\begingroup$ The null is that the accuracy of both methods is the same. $\endgroup$ – Gregg H Apr 27 '18 at 12:11
  • $\begingroup$ but the test is not on the accuracy it's on the entire classification.The McNemar's test is is applied to 2 × 2 contingency tables, true and false of the old classification and the true and false of the new classification $\endgroup$ – learneRS Apr 27 '18 at 14:18
  • $\begingroup$ Ok...but you ask in the question "if this accuracy improvement" is significant. If you want to test the accuracy, the 2x2 table (1 variable = the algorithm, 1 variable = correct or not); if you want to test the over all model results, then you would need a 2x2x2 table (1 variable for algorithm, 1 variable for original data, 1 variable for classification category). $\endgroup$ – Gregg H Apr 27 '18 at 15:38

@GreggH why 2x2x2 table? Can I just use 2x2?

For exemple using R: algo1<-c(rep('false',20),rep('true',50)) algo2<-c(rep('false',10),rep('true',60)) table(algo1,algo2) #the contingency table algo2 algo1 false true false 10 10 true 0 50 mcnemar.test(table(algo1,algo2))


McNemar's Chi-squared test with continuity correction data: table(algo1, algo2) McNemar's chi-squared = 8.1, df = 1, p-value = 0.004427

Also McNemar's test works only with 2x2 contingency tables (if i am not mistaken) here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.