How to plot ROC for multi-class classifier? One-vs-All or One-vs-One?

Question

I am trying to evaluate a multi-class classification model, say Random Forest, I built in R. So far, my approach has been looking at the ConfusionMatrix from the caret package and other results that are calculated by that function, but it gives me only a class wise result. I am also calculating micro and macro averages for estimating the overall accuracy for the model.

Now, when I am trying to plot the ROC curve, I have two options:

• One-vs-One approach: gives me ⁿC₂ combinations of ROC curves, which I am not sure how to interpret. The pROC package implements this method, suggested by Hand and Till because it supposedly gives a more accurate AUC.
• One-vs-All approach: gives me n ROC curves and their corresponding AUC. A reference for this I found in another similar question

Is there any preferred method for plotting the ROC in this scenario? If so, why?

And can this choice of it being either One-vs-One or One-vs-All be generalized to any multi-class model?

Answer 1

When you use roc curves, you are saying that

• misclassification costs are not the same for different types of mistakes. If they were, you would just optimize classification accuracy which would be the most adequate objective function and also more intuitive than all of the alternatives.
• you don't know the actual misclassification costs and therefore want to compare classification based on multiple cutoffs. If you did, you would just put the actual costs in the objective function to optimize.

If both these circumstances are given, you may look into roc curves. Though F-measure and kappa are worth considering as well.

The one vs. all approach reflects that some classes like $A$ may or may not be more costly to miss than other classes $B$ or $C$. Each roc curve sweeps over possible cutoffs for $A$ vs. ($B$ or $C$), $B$ vs. ($A$ or $C$) and the last one for $C$ vs. ($A$ or $B$)

The one vs. one approach is only necessary if for a given class $A$, it is also different to mistake an $A$ for a $B$ than to mistake an $A$ for a $C$. This is the most general case, but also the least intuitive.