I have a couple of classes and a classifier which tries to guess the classes.Given
- a dataset with instances $x_i$ together with $N$ classes where every instance $x_i$ belongs exactly to one class $y_i$
- a multiclass classifier
After the training and testing I basically have a listtable with the $y_i\rightarrow a_i$ wheretrue class $y_i$ is the real classes and the predicted class $a_i$ my machine predictionfor every instance $x_i$ in the test set. This match can beSo for every instance I have either a hitmatch (equal classes$y_i= a_i$) or a miss ($y_i\neq a_i$).
How can I evaluate the quality of the match ? The issue is that some real classes can have many members, i.e. many instances belong to it. Obviously if one real class has 50% of all data points belong to one class and my final classifier is 50% correct overall, I have gained nothing. I could have just as well made a trivial classifier which outputs that biggest class no matter what the input is.
Is there a standard method to estimate the quality of a classifier based on the known testing set results of matches and hits for each class? Maybe it's even important to distinguish matching rates for each particular class?
The simplest approach I can think of is to exclude the correct matches of the biggest class. What else?
