# Issues regarding cross-validation and metrics for comparing approaches in machine learning for image classification with imbalanced datasets

I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics:

• accuracy:
• macro-f1:
• weighted f1
• macro precision
• weight precision
• macro recall
• weighted recall
• macro auroc
• weighted auroc

Besides that, I'm also collecting the average of the same metrics for the top-5 and top-10 classes, regarding each metric itself (for example, the average accuracy in the top-5 classes regarding accuracy) and regarding the classes with the smaller number of instances (for example, the average accuracy in the top-5 classes with the smaller number os instances).

Do you recommend other interesting metrics in this context?

Besides that, I want to use n-fold cross-validation for evaluating the models. From what I understood, the idea is to split the complete dataset into n folds and iteratively use each fold as test data and the combination of the instances of the other folds as training. In each iteration, the idea is to collect the metrics and, at the end of the process, we report the average metric considering the metrics produced by each test fold. Right?

In this context, what is recommended as the best practices:

• Random cross-validation or stratified cross-validation (where each fold preserves the proportion of instances in the complete dataset)?
• Using random folds in each experiment or fixing a set of folds that are used in all the experiments (with all the classifiers)?
• Log/crossentropy loss and/or Brier score?
– Dave
Apr 8 at 0:23
• @Dave, sorry...Are these other metrics that you are suggesting for this context? Apr 8 at 0:32
• @Dave: Cross-entropy is almost inscrutable to explain classifier performance (what's the last time one used raw likelihood values to explain classifier performance?). Brier score, sure. Proper-scoring rule, with you (+1). Apr 8 at 0:38

3. Using random folds in each experiment vs fixing a set of folds is a good point to raise. I would suggest you fix the set of folds. It is quite trivial to implement (just set the seed before sampling/instantiating a sampling iterator) and ensures that we compare like for like. Again it should matter "too much", especially if we do repeated $$k$$-fold. On that last point, please see my answer in this thread: "Statistical significance when comparing two models for classification" it contains a number of references that will be relevant when comparing classifiers' performance.
• @EngrStudent: Yep, it is an option too (+1); I aimed to suggest metrics that do not rely on thresholding as I think it is more relevant during classifier comparisons but sure, Cohen's $k$ is another metric to consider. Apr 8 at 2:48