1
$\begingroup$

I am looking for some comprehensive instructions and ideally out of the box solutions (ideally for python) for evaluating different classifiers (which are already trained) for a multiclass classification problem on an unbalanced dataset.

To illustrate further: I have about a dozen classifiers that are trained on the same unbalanced dataset of a hand full of categories. Now I would like to

1) compare the classifiers against the ground truth:

How well do they perform on classifying on a per class basis (compared to a chance based model) and what is a sensible average of the per class performances?

2) compare the classifiers against each other:

Are they significantly different in what they classify data instances as? Are they significantly different in their overall performance (e.g. in accuracy per class)?

I looked into many test statistics now, some are

  • overall accuracy (bad for imbalanced datasets)
  • Cohen's kappa
  • Chi square goodness of fit
  • McNemar
  • AUROC
  • Brier score
  • Youden Index
  • Informedness
  • F-Score

I encountered different accounts whether these are suited for the imbalanced multiclass scenario and under which conditions they can be used, however. Most of the guides and explanations I read limited themselves to cases of binary classification.

I found the pycm package though, which computes many statistics (and most of the above), also for multiclass problems. But the documentation is kind of sparse, and I am not sure if it handles the unbalanced multiclass scenario correctly.

Now I am looking for some clear instructions on which tests I can apply to my case or how I need to format my data to be suited for some given test (I read about binarization of multiclass labes and "one vs all" a couple of times, for example, but these involved retraining the models (e.g. here), which is not an option for me.).

edit:

I am not asking about why accuracy is not a good metric. I am asking for which tests are suited for unbalanced multiclass.

Improve this question
$\endgroup$
27
  • 1
    $\begingroup$ Scoring rules all work on probabilistic predictions and actual outcome, by evaluating the predicted probability for a test sample to show the outcome actually observed. (The different scoring rules like the log score or the Brier score then proceed by transforming this probability in different ways.) Thus, they have no problems with unbalanced classes - as long as the class membership probability predictions are correct, the scoring rule will pick this up, no matter whether it's 1% or 50%. ... $\endgroup$
    – Stephan Kolassa
    Commented Apr 15, 2019 at 6:40
  • 1
    $\begingroup$ ... So scoring rules will always compare your predictions against reality, and you can compare the scores between models. (Usually, lower scores are better, but some people use the opposite convention - check the definition of the score you are using.) You can of course always look at the scores your models achieve on specific target classes and compare scores only on specific classes. ... $\endgroup$
    – Stephan Kolassa
    Commented Apr 15, 2019 at 6:42
  • 1
    $\begingroup$ ... I don't know of a way to make AUROC work for multi-class classification. Or ROC curves in general. The horizontal axis would need to turn into a $k-1$-dimensional space if you have $k$ classes, and you would evaluate over a simplex in that space. Anyway, AUROC is not very good for distinguishing models. Hope that helped! $\endgroup$
    – Stephan Kolassa
    Commented Apr 15, 2019 at 6:43
  • 1
    $\begingroup$ This post by Frank Harrell may be helpful. Or Gneiting & Katzfuss (2014). Or Merkle & Steyvers (2013). These two articles are mostly about numerical predictions, not classification, but everything holds with probabilistic predictions instead of predictive densities. $\endgroup$
    – Stephan Kolassa
    Commented Apr 15, 2019 at 6:53
  • 2
    $\begingroup$ In each bootstrap replicate, divide your sample randomly into a training and a testing set. Fit models to the training set, probabilistically predict the test set, record the value of the scoring rule. Do this 1,000 times. You get 1,000 bootstrap replicates of the scoring rule value of each model. Then you can see whether, e.g., one model consistently outperforms another one, by having a lower scoring rule value on 95% of the bootstrap replicates. $\endgroup$
    – Stephan Kolassa
    Commented Apr 15, 2019 at 15:06

0

Reset to default