Issues regarding cross-validation and metrics for comparing approaches in machine learning for image classification with imbalanced datasets

Question

I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics:

• accuracy:
• macro-f1:
• weighted f1
• macro precision
• weight precision
• macro recall
• weighted recall
• macro auroc
• weighted auroc

Besides that, I'm also collecting the average of the same metrics for the top-5 and top-10 classes, regarding each metric itself (for example, the average accuracy in the top-5 classes regarding accuracy) and regarding the classes with the smaller number of instances (for example, the average accuracy in the top-5 classes with the smaller number os instances).

Do you recommend other interesting metrics in this context?

Besides that, I want to use n-fold cross-validation for evaluating the models. From what I understood, the idea is to split the complete dataset into n folds and iteratively use each fold as test data and the combination of the instances of the other folds as training. In each iteration, the idea is to collect the metrics and, at the end of the process, we report the average metric considering the metrics produced by each test fold. Right?

In this context, what is recommended as the best practices:

• Random cross-validation or stratified cross-validation (where each fold preserves the proportion of instances in the complete dataset)?
• Using random folds in each experiment or fixing a set of folds that are used in all the experiments (with all the classifiers)?

Answer 1

I think your understanding is mostly correct. :)

Regarding your particular questions:

1. Brier score (the mean squared difference between the predicted probability and the actual outcome) and AUC-PR (Area Under the Curve for Precision-Recall; sometimes called average precision score) are two more metrics one could consider for a classification problem. They are well-suited for "imbalanced" tasks too.
2. Random cross-validation vs. stratified cross-validation should have little impact if the sample is large enough. Via stratification, we will ensure that each fold has very similar (usually identical) class proportions. Unless we see large performance metric fluctuation in per fold performance than can be attributed to sampling variation, stratification isn't that beneficial. So to begin with, do not do it, allow for the sampling variation to be part of the validation schema and if that sampling variation becomes too much of an issue, revisit the use of stratification.
3. Using random folds in each experiment vs fixing a set of folds is a good point to raise. I would suggest you fix the set of folds. It is quite trivial to implement (just set the seed before sampling/instantiating a sampling iterator) and ensures that we compare like for like. Again it should matter "too much", especially if we do repeated $k$-fold. On that last point, please see my answer in this thread: "Statistical significance when comparing two models for classification" it contains a number of references that will be relevant when comparing classifiers' performance.