0
$\begingroup$

full disclosure:

I did a semi-cross post of this question due to low traffic. Once I get an answer on any of the two questions, I will link the answer back to the respective other.


tl;dr

For multiclass classifiers, do you need to apply McNemar or Cochran's Q to determine, whether two classifiers are significantly different in how they categorize the same data?


I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Examples:

Machine Learning Model Comparison - Doubts applying statistical tests

http://web.cs.iastate.edu/~honavar/dietterich98approximate.pdf

However, I am not sure if these sources assumed binary classifiers. I then found out about Cochran's Q, which is some sort of generalization of McNemar's test for $n \times m$ contingency tables where $n$ and $m$ can be greater than 2.

I would like to know which of the two tests I need to or can apply to the case of multiclass classifiers.

Let me give you an example. For that let's generate a bit of random data

>>> # number of categories
... k = 4
>>> 
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)

Now, given this data, can I apply McNemar?

>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>> 
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>> 
>>> print(table)
[[559 186]
 [186  69]]
>>> print(mcnemar(table))
pvalue      1.0
statistic   186.0

Or do I have to use Cochran's Q, because I don't have a dichotomous variable?

>>> table = np.zeros(shape=(k, k))
>>> 
>>> for p1, p2 in zip(preds1, preds2):
...     table[p1, p2]+=1
... 
>>> print(table)
[[62. 73. 58. 71.]
 [61. 64. 77. 63.]
 [64. 59. 55. 51.]
 [50. 63. 62. 67.]]
>>> print(cochrans_q(table))
df          3
pvalue      0.3916251762710877
statistic   3.0

  • I used the statsmodels library for the tests.

  • My classes are imbalanced... I don't know if that plays a role here.


EDIT

As I did not find a bowker test for python, I wrote it myself:

from scipy import stats import numpy as np

class DotDict(dict):
    __getattr__ = dict.__getitem__
    __setattr__ = dict.__setitem__
    __delattr__ = dict.__delitem__

    def __init__(self, dct):
        for key, value in dct.items():
            if hasattr(value, 'keys'):
                value = DotDict(value)
            self[key] = value

def div0( a, b ):
    with np.errstate(divide='ignore', invalid='ignore'):
        c = np.true_divide( a, b )
        c[ ~ np.isfinite( c )] = 0  # -inf inf NaN
    return c

def bowker(table):

    num_cols, num_rows = table.shape
    tril = np.tril(table, k=-1)
    triu = np.triu(table, k=1)
    numer = (tril - triu.T)**2
    denom = (tril + triu.T)
    quot = div0(numer, denom) 
    statistic = np.sum(quot)
    df = int(num_rows * (num_rows -1) / 2)
    pvalue = stats.chi2.sf(statistic, df)
    return DotDict({'pvalue' : pvalue, 'statistic' : statistic, 'df' : df})

You can verify the code tobe correct by comparing the results of this R code

mat = as.table(rbind(c( 7, 17, 8), 
                     c(14, 8, 9),
                     c(11, 15, 11)))


mcnemar.test(mat, correct=FALSE)

mat = as.table(rbind(c(20, 10, 5), 
                     c(3, 30, 15),
                     c(0, 5, 40)))


mcnemar.test(mat, correct=FALSE)

to this python code:

table = np.array(
[[20, 10, 5], 
 [3, 30, 15],
 [0, 5, 40]]
)

print(bowker(table))

table = np.array(
[[ 7, 17, 8], 
 [14, 8, 9],
 [11, 15, 11]]
)

print(bowker(table))

which will give the same results.

$\endgroup$
  • 1
    $\begingroup$ I think you have misundwestood in what way Cochran's Q is a generalization of McNemar. Cochran's Q (at least in any form I've seen it) works only on binary outcomes. McNemar can be generalized to m x n. This is sometimes called McNemar-Bowker, or other names. The default mcnemar.test in R handles tables larger than 2 x 2. I can't comment on the Python functions or what's appropriate for multiclass classifiers. $\endgroup$ – Sal Mangiafico Apr 18 at 16:33
  • $\begingroup$ That is already helpful! Hmm. I compared the results of the R mcnemar test against the statsmodels mcnemar test, which also accepts $n \times n$ where $n > 2$ tables. But the results differed... Therefore I implemented the Bowker test myself and added it to my post. $\endgroup$ – lo tolmencre Apr 18 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.