# full disclosure:

I did a semi-cross post of this question due to low traffic. Once I get an answer on any of the two questions, I will link the answer back to the respective other.

# tl;dr

For multiclass classifiers, do you need to apply McNemar or Cochran's Q to determine, whether two classifiers are significantly different in how they categorize the same data?

I need to determine whether a number of classifiers are pairwise significantly different in their predictions. I found several sources mentioning McNemar as suited for this. Examples:

Machine Learning Model Comparison - Doubts applying statistical tests

http://web.cs.iastate.edu/~honavar/dietterich98approximate.pdf

However, I am not sure if these sources assumed binary classifiers. I then found out about Cochran's Q, which is some sort of generalization of McNemar's test for $$n \times m$$ contingency tables where $$n$$ and $$m$$ can be greater than 2.

I would like to know which of the two tests I need to or can apply to the case of multiclass classifiers.

Let me give you an example. For that let's generate a bit of random data

>>> # number of categories
... k = 4
>>>
>>> # random data representing the ground truth in k categories
... ground_truth = np.random.randint(0,k,1000)
>>> # random data representing predictions by two different classifiers
... preds1 = np.random.randint(0,k,1000)
>>> preds2 = np.random.randint(0,k,1000)


Now, given this data, can I apply McNemar?

>>> # binary arrays coding for whether a prediction did match with the ground truth
... results1 = preds1 == ground_truth
>>> results2 = preds2 == ground_truth
>>>
>>> table = np.bincount(2 * (results1) + (results2), minlength=2*2).reshape(2, 2)
>>>
>>> print(table)
[[559 186]
[186  69]]
>>> print(mcnemar(table))
pvalue      1.0
statistic   186.0


Or do I have to use Cochran's Q, because I don't have a dichotomous variable?

>>> table = np.zeros(shape=(k, k))
>>>
>>> for p1, p2 in zip(preds1, preds2):
...     table[p1, p2]+=1
...
>>> print(table)
[[62. 73. 58. 71.]
[61. 64. 77. 63.]
[64. 59. 55. 51.]
[50. 63. 62. 67.]]
>>> print(cochrans_q(table))
df          3
pvalue      0.3916251762710877
statistic   3.0


• I used the statsmodels library for the tests.

• My classes are imbalanced... I don't know if that plays a role here.

# EDIT

As I did not find a bowker test for python, I wrote it myself:

from scipy import stats import numpy as np

class DotDict(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__

def __init__(self, dct):
for key, value in dct.items():
if hasattr(value, 'keys'):
value = DotDict(value)
self[key] = value

def div0( a, b ):
with np.errstate(divide='ignore', invalid='ignore'):
c = np.true_divide( a, b )
c[ ~ np.isfinite( c )] = 0  # -inf inf NaN
return c

def bowker(table):

num_cols, num_rows = table.shape
tril = np.tril(table, k=-1)
triu = np.triu(table, k=1)
numer = (tril - triu.T)**2
denom = (tril + triu.T)
quot = div0(numer, denom)
statistic = np.sum(quot)
df = int(num_rows * (num_rows -1) / 2)
pvalue = stats.chi2.sf(statistic, df)
return DotDict({'pvalue' : pvalue, 'statistic' : statistic, 'df' : df})


You can verify the code tobe correct by comparing the results of this R code

mat = as.table(rbind(c( 7, 17, 8),
c(14, 8, 9),
c(11, 15, 11)))

mcnemar.test(mat, correct=FALSE)

mat = as.table(rbind(c(20, 10, 5),
c(3, 30, 15),
c(0, 5, 40)))

mcnemar.test(mat, correct=FALSE)


to this python code:

table = np.array(
[[20, 10, 5],
[3, 30, 15],
[0, 5, 40]]
)

print(bowker(table))

table = np.array(
[[ 7, 17, 8],
[14, 8, 9],
[11, 15, 11]]
)

print(bowker(table))


which will give the same results.

• I think you have misundwestood in what way Cochran's Q is a generalization of McNemar. Cochran's Q (at least in any form I've seen it) works only on binary outcomes. McNemar can be generalized to m x n. This is sometimes called McNemar-Bowker, or other names. The default mcnemar.test in R handles tables larger than 2 x 2. I can't comment on the Python functions or what's appropriate for multiclass classifiers. – Sal Mangiafico Apr 18 '19 at 16:33
• That is already helpful! Hmm. I compared the results of the R mcnemar test against the statsmodels mcnemar test, which also accepts $n \times n$ where $n > 2$ tables. But the results differed... Therefore I implemented the Bowker test myself and added it to my post. – lo tolmencre Apr 18 '19 at 19:09