1
$\begingroup$

I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics:

  • accuracy:
  • macro-f1:
  • weighted f1
  • macro precision
  • weight precision
  • macro recall
  • weighted recall
  • macro auroc
  • weighted auroc

Besides that, I'm also collecting the average of the same metrics for the top-5 and top-10 classes, regarding each metric itself (for example, the average accuracy in the top-5 classes regarding accuracy) and regarding the classes with the smaller number of instances (for example, the average accuracy in the top-5 classes with the smaller number os instances).

Do you recommend other interesting metrics in this context?

Besides that, I want to use n-fold cross-validation for evaluating the models. From what I understood, the idea is to split the complete dataset into n folds and iteratively use each fold as test data and the combination of the instances of the other folds as training. In each iteration, the idea is to collect the metrics and, at the end of the process, we report the average metric considering the metrics produced by each test fold. Right?

In this context, what is recommended as the best practices:

  • Random cross-validation or stratified cross-validation (where each fold preserves the proportion of instances in the complete dataset)?
  • Using random folds in each experiment or fixing a set of folds that are used in all the experiments (with all the classifiers)?
$\endgroup$
3
  • 2
    $\begingroup$ Log/crossentropy loss and/or Brier score? $\endgroup$
    – Dave
    Apr 8 at 0:23
  • $\begingroup$ @Dave, sorry...Are these other metrics that you are suggesting for this context? $\endgroup$
    – Zaratruta
    Apr 8 at 0:32
  • 1
    $\begingroup$ @Dave: Cross-entropy is almost inscrutable to explain classifier performance (what's the last time one used raw likelihood values to explain classifier performance?). Brier score, sure. Proper-scoring rule, with you (+1). $\endgroup$
    – usεr11852
    Apr 8 at 0:38

1 Answer 1

1
$\begingroup$

I think your understanding is mostly correct. :)

Regarding your particular questions:

  1. Brier score (the mean squared difference between the predicted probability and the actual outcome) and AUC-PR (Area Under the Curve for Precision-Recall; sometimes called average precision score) are two more metrics one could consider for a classification problem. They are well-suited for "imbalanced" tasks too.
  2. Random cross-validation vs. stratified cross-validation should have little impact if the sample is large enough. Via stratification, we will ensure that each fold has very similar (usually identical) class proportions. Unless we see large performance metric fluctuation in per fold performance than can be attributed to sampling variation, stratification isn't that beneficial. So to begin with, do not do it, allow for the sampling variation to be part of the validation schema and if that sampling variation becomes too much of an issue, revisit the use of stratification.
  3. Using random folds in each experiment vs fixing a set of folds is a good point to raise. I would suggest you fix the set of folds. It is quite trivial to implement (just set the seed before sampling/instantiating a sampling iterator) and ensures that we compare like for like. Again it should matter "too much", especially if we do repeated $k$-fold. On that last point, please see my answer in this thread: "Statistical significance when comparing two models for classification" it contains a number of references that will be relevant when comparing classifiers' performance.
$\endgroup$
2
  • 2
    $\begingroup$ I like cohen’s kappa. $\endgroup$ Apr 8 at 2:05
  • 2
    $\begingroup$ @EngrStudent: Yep, it is an option too (+1); I aimed to suggest metrics that do not rely on thresholding as I think it is more relevant during classifier comparisons but sure, Cohen's $k$ is another metric to consider. $\endgroup$
    – usεr11852
    Apr 8 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.