I have a fairly rubbish classifier for a task assigning samples to one of the classes {i, e, l, g}. I would like to know whether it's actually better than random.

All the discussion that I can find is about telling whether one classifier is significantly better than another, which seems to me to be a different question. My initial thought was that using kappa (Python implementation of Cohen's basic kappa) to compare the ground truth and the classifier as though they were two people annotating reality would work. Doing this gives me a kappa score of 0.06, which suggests that my classifier is barely better than random, but the notes on SciKit-Learn Cohen’s kappa say "This measure is intended to compare labelings by different human annotators, not a classifier versus a ground truth." So my questions are (i) why shouldn't I use kappa for this purpose and (ii) what should I use?

(I know that there's a similar question, Cohen's Kappa Classifier vs. Groundtruth, on Cross Validated, but the answers there to (i) don't really say what's wrong with doing it, and there isn't an answer to (ii))

  • $\begingroup$ If you have $N$ classes to choose from, and you guess uniformly at random, the probability of picking any single class would be $\frac{1}{N}$. Therefore, a simple way to detect way if your classifier is performing better than random is to check if its accuracy is greater than $\frac{1}{N}$. A more meaningful way would be to compute its Matthews correlation coefficient. $\endgroup$
    – mhdadk
    Jul 22 '21 at 10:05
  • $\begingroup$ My problem is that I need to know whether the difference between what my classifier produces and $1/N$ is significant. My test data is evenly split between 4 classes, so a random classifier would score about 25%. You wouldn't expect it to score exactly 25%, you'd expect it to score about 25%. So if my classifier scores 32% on a test set of 300 cases, should I take that as being significantly better than chance or not? (using the Matthews correlation coefficient is only going to be useful if the test cases aren't uniformly split between classes, but mine are). $\endgroup$ Jul 23 '21 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.