When comparing performance of two binary classifiers using McNemar test, should the two confusion matrices of the models be based on the training set, the validation set or even a second validation set which was not used for parameter tuning? What is the justification for the correct approach?

  • $\begingroup$ no, it's a legitimate question. I read some papers on this like citeseerx.ist.psu.edu/viewdoc/…. They use the term "test data" but it is ambiguous. $\endgroup$ – salvador Jan 17 '17 at 17:47
  • $\begingroup$ Ah. In that case, you should at least use a validation set, or better yet a second validation set, as you described. You can also cross-validate. $\endgroup$ – Zach Jan 19 '17 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.