Statistical hypothesis test for classification accuracy

I just want to make sure I'm doing things right!

I created 2 algorithms that classify my data into 2 groups. The first one gives me an accuracy of 76.66% (sensitivity=0.76, specificity=0.78). The second one: accuracy 81.33% (sensitivity=0.86, specificity=0.76)

So i wanted to know if this improvement is "statistically significant". I used McNemar's test (statistical test used on paired nominal data. It is applied to 2 × 2 contingency tables).

Is it a "good" way to proceed? if not which test should I choose?

• Yes, a 2x2 contingency table would be appropriate to assess the null $H_0: \text{accuracy of both methods the same}$ – Gregg H Apr 25 '18 at 13:18
• Thank you! What would be the null hypothesis? – learneRS Apr 27 '18 at 7:40
• The null is that the accuracy of both methods is the same. – Gregg H Apr 27 '18 at 12:11
• but the test is not on the accuracy it's on the entire classification.The McNemar's test is is applied to 2 × 2 contingency tables, true and false of the old classification and the true and false of the new classification – learneRS Apr 27 '18 at 14:18
• Ok...but you ask in the question "if this accuracy improvement" is significant. If you want to test the accuracy, the 2x2 table (1 variable = the algorithm, 1 variable = correct or not); if you want to test the over all model results, then you would need a 2x2x2 table (1 variable for algorithm, 1 variable for original data, 1 variable for classification category). – Gregg H Apr 27 '18 at 15:38

@GreggH why 2x2x2 table? Can I just use 2x2?

For exemple using R:  algo1<-c(rep('false',20),rep('true',50)) algo2<-c(rep('false',10),rep('true',60)) table(algo1,algo2) #the contingency table algo2 algo1 false true false 10 10 true 0 50 mcnemar.test(table(algo1,algo2))

Result:

McNemar's Chi-squared test with continuity correction data: table(algo1, algo2) McNemar's chi-squared = 8.1, df = 1, p-value = 0.004427 

Also McNemar's test works only with 2x2 contingency tables (if i am not mistaken) here