Then two classifiers are evaluated on the same dataset, McNemar's test could be used to compare their overall classification accuracy, telling if the difference between them are significant. But what if we like to compare them separately on type I and type II errors, quantifying the difference on both precision and recall? It seems like McNemar's test could not be used at least for precision because of the randomness of the denominator.

What are the suitable statistical tests for this case?


While it doesn't directly answer your question (how to compare precision), the problem of the denominator changing in precision is why specificity is often used instead of precision when looking at type I and type II errors.

From http://en.wikipedia.org/wiki/Sensitivity_and_specificity:

Specificity (sometimes called the true negative rate) measures the proportion of negatives which are correctly identified as such (e.g. the percentage of healthy people who are correctly identified as not having the condition).

You could then apply McNemar's test to sensitivity (which is identical to recall) and specificity separately.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.