Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Given

  • a dataset with instances $x_i$ together with $N$ classes where every instance $x_i$ belongs exactly to one class $y_i$
  • a multiclass classifier

After the training and testing I basically have a table with the true class $y_i$ and the predicted class $a_i$ for every instance $x_i$ in the test set. So for every instance I have either a match ($y_i= a_i$) or a miss ($y_i\neq a_i$).

How can I evaluate the quality of the match ? The issue is that some classes can have many members, i.e. many instances belong to it. Obviously if 50% of all data points belong to one class and my final classifier is 50% correct overall, I have gained nothing. I could have just as well made a trivial classifier which outputs that biggest class no matter what the input is.

Is there a standard method to estimate the quality of a classifier based on the known testing set results of matches and hits for each class? Maybe it's even important to distinguish matching rates for each particular class?

The simplest approach I can think of is to exclude the correct matches of the biggest class. What else?

share|improve this question
    
I am not sure if I understand the question correctly. Do you know the Confusion Matrix and derived measures ? Is this the answer to your question ? Or do you refer to something more complicated ? –  steffen Nov 27 '12 at 9:27
    
I think this is the source of my confusion: In the first paragraph you state ..where yi is the real classes and...: Do you mean that an instance $x_i$ can belong to / has more than one class ? Or does every $x_i$ belongs to / has exactly one class ? Can you please clarify ? –  steffen Nov 27 '12 at 9:30
    
@steffen: I've seen the confusion matrix. In my particular case I have 4 classes. So I'm not sure which derived measures can be used and would make sense. Each $x_i$ belongs to only one class. However there are more than two possible classes overall $i\in [1,\cdots,N]$. –  Gerenuk Nov 27 '12 at 13:24
    
@steffen Those derived measures are primarily applicable to binary classification, whereas this question explicitly is dealing with more than two classes. This then requires a modified understanding of terms like "true positive." –  Michael McGowan Nov 27 '12 at 14:36
    
@MichaelMcGowan I have asked the OP for clarification and afterwards performed an edit to explicitly reflect the multiclass issue, which was not obvious before the edit (IMHO). –  steffen Nov 27 '12 at 14:47
show 1 more comment

2 Answers

up vote 4 down vote accepted

Like binary classification, you can use the empirical error rate for estimating the quality of your classifier. Let $g$ be a classifier, and $x_i$ and $y_i$ be respectively an example in your data base and its class. $$err(g) = \frac{1}{n} \sum_{i \leq n} \mathbb{1}_{g(x_i) \neq y_i}$$ As you said, when the classes are unbalanced, the baseline is not 50% but the proportion of the bigger class. You could add a weight on each class to balance the error. Let $W_y$ be the weight of the class $y$. Set the weights such that $\frac{1}{W_y} \sim \frac{1}{n}\sum_{i \leq n} \mathbb{1}_{y_i = y}$ and define the weighted empirical error $$err_W(g) = \frac{1}{n} \sum_{i \leq n} W_{y_i} \mathbb{1}_{g(x_i) \neq y_i}$$

As Steffen said, the confusion matrix could be a good way to estimate the quality of a classifier. In the binary case, you can derive some measure from this matrix such as sensitivity and specificity, estimating the capability of a classifier to detect a particular class. The source of error of a classifier might be in a particular way. For example a classifier could be too much confident when predicting a 1, but never say wrong when predicting a 0. Many classifiers can be parametrized to control this rate (false positives vs false negatives), and you are then interested in the quality of the whole family of classifier, not just one. From this you can plot the ROC curve, and measuring the area under the ROC curve give you the quality of those classifiers.

ROC curves can be extended for your multiclass problem. I suggest you to read the answer of this thread.

share|improve this answer
add comment

To evaluate multi-way text classification systems, I use micro- and macro-averaged F1 (F-measure). The F-measure is essentially a weighted combination of precision and recall that. For binary classification, the micro and macro approaches are the same, but, for the multi-way case, I think they might help you out. You can think of Micro F1 as a weighted combination of precision and recall that gives equal weight to every document, while Macro F1 gives equal weight to every class. For each, the F-measure equation is the same, but you calculate precision and recall differently:

$$F = \frac{(\beta^{2} + 1)PR}{\beta^{2}P+R},$$

where $\beta$ is typically set to 1. Then,

$$P_{micro}=\frac{\sum^{|C|}_{i=1}TP_{i}}{\sum^{|C|}_{i=1}TP_{i}+FP_{i}}, R_{micro}=\frac{\sum^{|C|}_{i=1}TP_{i}}{\sum^{|C|}_{i=1}TP_{i}+FN_{i}}$$

$$P_{macro}=\frac{1}{|C|}\sum^{|C|}_{i=1}\frac{TP_{i}}{TP_{i}+FP_{i}}, R_{macro}=\frac{1}{|C|}\sum^{|C|}_{i=1}\frac{TP_{i}}{TP_{i}+FN_{i}}$$

where $TP$ is True Positive, $FP$ is False Positive, $FN$ is False Negative, and $C$ is class.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.