How to determine whether a classifier is significantly better than random guessing? I have a classifier Y that selects between three categories: A, B and C.
I need to be able to quantitatively prove that my model is better (and by how much?), than a random classifier R that randomly picks between categories A, B and C.
I intend to proceed as follows:


*

*Generate classifications using classifier R

*Generate a confusion matrix for output of classifier R

*Generate classifications using classifier Y

*Generate a confusion matrix for output of classifier Y
However, having generated the two confusion matrices above, I'm not sure how to use them to solve my problem.
The "intuition" behind using the confusion matrices is that I can "visually" check and "compare" the sensitivity, specificity etc between the models etc.
I would like to be able to use the confusion matrices (if possible) to do some test the null hypothesis that the BookMaker Informedness of Y is no better than R
Can anyone help with how I can test this hypothesis, given data from the two confusion matrices?
 A: *

*If you care mostly about accuracy (which I deduce from the fact that you are discussing confusion matrices), a meaningful benchmark is not a random classifier, but one that classifies everything as the most common class in the training set. It will always have an accuracy at least as high as random classification (though not sensitivity and specificity).

*This is one reason why accuracy is not a good evaluation measure for classifiers. Neither are confusion matrices, nor any measures derived from them, like sensitivity or specificity. I give arguments here.

*Instead, use probabilistic predictions and assess these using proper scoring rules. See the link above for more information.

*To assess whether one classifier is better than a "benchmark" one (e.g., one that always assigns a probability of 1 to the majority class and a probability of 0 to all others), you need to derive the distribution of the score of the benchmark. You can get this by bootstrapping: resample the test set, apply the benchmark to the resampled set, note the score. Repeat this many times. You now have the distribution you are looking for. Check how many of these resampled scores are smaller (since scores are better when smaller) than the score of your candidate classifier.

*If you insist on using confusion matrices and summarizing them using a single number like $J$, you can do the same: resample the test set, apply your benchmark (random classification, or, as I propose above, classify-everything-into-the-majority-class) to get a confusion matrix, calculate $J$ (or whatever). Do this many times. You now have the distribution of $J$ for your benchmark. Check how many of these resampled $J$s exceed the $J$ of your candidate method on the entire test set.
Good luck!
