Compare classification performance of two heuristics

Question

I want to compare the classification performance of two heuristics: $h_0$ and $h_1$. I have around 700 samples for which I know the output class $c$ and the results of the two heuristics $c_{\textit{h_0}}, c_{\textit{h_1}}$. That is, for each sample I have $\text{sample}_{\textit{id}}, c, c_{\textit{h_0}}, c_{\textit{h_1}}$.

$h_0$ has around ~71% correct predictions of the output class and $h_1$ has around ~77% correct. What I want to calculate now is whether this performance difference is just by chance or if the prediction performance of $h_1$ is statistical significant better than that of $h_0$.

I have found Fisher's exact test, but I don't know whether I can just put my data in there like the following matrix to calculate a p-value.

$$ \begin{matrix} & h_0 & h_1 \\ \text{correct prediction} & 497 & 539 \\ \text{wrong prediction} & 203 & 161 \end{matrix} $$

I don't know if I can calculate the p-value like that or if I'm completely on the wrong track. I would really appreciate it if someone could point me in the right direction.

Answer 1

You have matched pairs. That is, you (ultimately) have a correct / wrong classification for each sample. (You will need to get this status for each sample to validly assess the results.) As a result of this fact (among others), Fisher's exact test is not appropriate for your data.

Instead, you should format your table so that each count indexes the status of a given matched pair. In other words, you would enumerate the number of pairs for which both classifications were correct, only $h_0$ was correct, only $h_1$ was correct, and where both classifications were wrong. In the end, the sum of the counts in the table will be half of your current table. For example, your table might end up looking like this (totally made up):

tab = as.table(matrix(c(400, 150,
                         50, 100), nrow=2, byrow=2))
rownames(tab) <- colnames(tab) <- c("correct", "wrong")
names(dimnames(tab)) = c("h0", "h1")
tab
#          h1
# h0        correct wrong
#   correct     400   150
#   wrong        50   100

You can assess if the heuristics differ in predictive accuracy using McNemar's test. (I have explained McNemar's test here and here.)

mcnemar.test(tab)
#  McNemar's Chi-squared test with continuity correction
# 
# data:  tab
# McNemar's chi-squared = 49.005, df = 1, p-value = 2.553e-12

---

Let me note that there may be a better way to evaluate these heuristics. Many models (e.g., logistic regression) will output a continuous value, which people then dichotomize to generate the predicted classifications. If your case is like this, you would do better to compare the outputted continuous values, rather than the classifications. Two possibilities would be to compare the area under the Receiver Operating Characteristic (ROC) curve (see here and here, e.g.), or to compare a proper score (such as the Brier score).