There is no single perfect solution to comparing multi class classification performances on basis of their individual classes - but there are different approaches that might be appropriate for your problem. Your approach of looking at the diagonal of the confusion matrix does conceptually make sense, but likely is more complicated than necessary, as the confusion matrix still holds much details (how classes are confused - which is good to know but complicated to automatically select a best model from).
To name some other ways of selecting a model by looking at results over individual classes: you could use multi class ROC curves (see e.g. this answer) and multi class AUC/AUROC (see e.g. multiclass.roc in pROC). The latter gives you a single performance metric value per model which can be used to decide on the single best model if necessary.
I usually end up computing one or more of those: a) all ROC curves for all 1-vs-all problems that are part of my $N$ class problem, and using those to look at b) the distribution of the AUCs over all classes, c) the distribution of EERs over all classes, c) and distribution of the true positive and false positive rates over all classes. The more models I have to evaluate, the more I usually automate this process. For example, for thousands of models I will compute only a single, scalar metric value which allows me to filter bad performing models, which will leave me with a smaller amount of models to look at in detail. Here's a simple example of how this could look like:
At first we create a model that gives some example data and is able to compute class probabilities:
library(caret)
library(pROC)
m <- train(iris[,1:2], iris[,5], method = 'lda', metric = 'Kappa', trControl = trainControl(method = 'repeatedcv', number = 10, repeats = 20, classProbs = T, savePredictions = T))
str(m$pred)
Next we calculate all ROC curves for all 1-vs-all problems part of our $N$ class problem:
rocs <- llply(unique(m$pred$obs), function(cls) {
roc(response = m$pred$obs==cls, predictor = m$pred[,as.character(cls)])
})
# for curiosity: look at the ROC curves
plot(rocs[[1]])
l_ply(rocs[-1], plot, add=T)
Using those ROC curves we can calculate all AUC for all classes. Those represent the models power per individual class, and its mean, median, or similar could already be seen as metric for the overall power:
aucs <- laply(rocs, function(x) as.numeric(x$auc))
summary(aucs)
The same is true for the EER per class:
eers <- laply(rocs, function(x) x$sensitivities[which.min(abs(x$sensitivities-x$specificities))])
summary(eers)
We can also look at the true positive and true negative rate per class - which is more related to your idea of looking at the diagonal of the confusion matrix than the previous approaches. The mean or median true positive and true negative rate (or the mean or median of those means/medians if you want to have a single value) could be used to derive the overall power of your model as well:
conf <- ldply(unique(m$pred$obs), function(cls) data.frame(tp=sum(m$pred$pred[m$pred$obs==cls]==cls),
fn=sum(m$pred$pred[m$pred$obs==cls]!=cls),
tn=sum(m$pred$pred[m$pred$obs!=cls]!=cls),
fp=sum(m$pred$pred[m$pred$obs!=cls]==cls)))
conf$tpr <- with(conf, tp/(tp+fn))
conf$tnr <- with(conf, tn/(tn+fp))
summary(conf[,5:6])
Bottom line: in general I would use such approaches to filter models you for sure don't want to look at - but for the remaining handful of models, looking at the confusion matrix usually is a good idea, as it still holds much information about what is going on for them.
