I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics:
- weighted f1
- macro precision
- weight precision
- macro recall
- weighted recall
- macro auroc
- weighted auroc
Besides that, I'm also collecting the average of the same metrics for the top-5 and top-10 classes, regarding each metric itself (for example, the average accuracy in the top-5 classes regarding accuracy) and regarding the classes with the smaller number of instances (for example, the average accuracy in the top-5 classes with the smaller number os instances).
Do you recommend other interesting metrics in this context?
Besides that, I want to use n-fold cross-validation for evaluating the models. From what I understood, the idea is to split the complete dataset into n folds and iteratively use each fold as test data and the combination of the instances of the other folds as training. In each iteration, the idea is to collect the metrics and, at the end of the process, we report the average metric considering the metrics produced by each test fold. Right?
In this context, what is recommended as the best practices:
- Random cross-validation or stratified cross-validation (where each fold preserves the proportion of instances in the complete dataset)?
- Using random folds in each experiment or fixing a set of folds that are used in all the experiments (with all the classifiers)?