What is the best way to compute classifier performance metrics given a confusion matrix?

Question

Let's say I compute a confusion matrix, in the sense defined here: http://www.gabormelli.com/RKB/Confusion_Matrix

I could easily compute the number of True Negatives (TN), True Positives (TP), False Negatives (FN) and False Positives (FP) but I feel it would be a bit awkward: in my case, all objects belong to at least one class: so a misclassification (e.g. my classifier put an actual "A" in a "B" bin) is not only a FP, it is a FN in the same time. And what if an actual "A+B" object is classified as "B" only: in my current study, it's definitely better than classified in "C" or not classified at all. I have other examples like this...

As you could guess, for me, those are confusing matrix.

Answer 1

Although it's not stated directly in your question, I guess you have a multiclass classification problem. As always, there are many performance metrics available, each emphasizes another property of the classifier. My favourite generic metric for this type of problems is multiclass $F1$ metric.

Let's say for a data point $x_i$ the set of true labels is $S_i$, and your classifier output is set of labels $C_i$. Then :

F1 measure on that example is $\frac{2 \cdot |S_i \cap C_i|}{|S_i| + |C_i|}$

And F1 measure on the whole data set is :

$\frac{1}{M} \cdot \sum_{i=1}^{M} \frac{2 \cdot |S_i \cap C_i|}{|S_i| + |C_i|}$

Notice it's 1 iff the classifier is perfect, and drops if the classifier makes either FP or FN mistakes (it dislikes FN mistakes a little more).