Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Let's say I compute a confusion matrix, in the sense defined here: http://www.gabormelli.com/RKB/Confusion_Matrix

I could easily compute the number of True Negatives (TN), True Positives (TP), False Negatives (FN) and False Positives (FP) but I feel it would be a bit awkward: in my case, all objects belong to at least one class: so a misclassification (e.g. my classifier put an actual "A" in a "B" bin) is not only a FP, it is a FN in the same time. And what if an actual "A+B" object is classified as "B" only: in my current study, it's definitely better than classified in "C" or not classified at all. I have other examples like this...

As you could guess, for me, those are confusing matrix.

share|improve this question
add comment

1 Answer

Although it's not stated directly in your question, I guess you have a multiclass classification problem. As always, there are many performance metrics available, each emphasizes another property of the classifier. My favourite generic metric for this type of problems is multiclass $F1$ metric.

Let's say for a data point $x_i$ the set of true labels is $S_i$, and your classifier output is set of labels $C_i$. Then :

F1 measure on that example is $\frac{2 \cdot |S_i \cap C_i|}{|S_i| + |C_i|}$

And F1 measure on the whole data set is :

$\frac{1}{M} \cdot \sum_{i=1}^{M} \frac{2 \cdot |S_i \cap C_i|}{|S_i| + |C_i|}$

Notice it's 1 iff the classifier is perfect, and drops if the classifier makes either FP or FN mistakes (it dislikes FN mistakes a little more).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.