2
$\begingroup$

I'm working with an imbalanced multi class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifierusing a randomized search and a bayesian search.

For now I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for the parameter tuning?

I found that for example recall_micro yields the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant? Can a different metric lead to different results? If yes, which metric should I use?

$\endgroup$

1 Answer 1

3
$\begingroup$

Yes, the scoring is relevant.

Hyperparameter tuning is done by ranking hyperparameter sets and choosing the best one. The best one here is identified by a scoring metric. Ideally you would want the scoring metric to be identical with the final evaluation metric.

For an imbalanced multiclass dataset I would recommend to use average classwise accuracy (mean of diagonal of the normalized confusion matrix), since it is not biased towards the class with the highest number of samples.

$\endgroup$
5
  • $\begingroup$ Ok thanks! Why do for example recall_micro and accuracy lead to the same results when I do hyperparameter tuning? $\endgroup$
    – Christian
    Apr 26, 2018 at 11:50
  • $\begingroup$ That is dataset dependent. If I sort all humans by (a) feet size and (b) body size - why is the first person the same in both cases (lets just assume it is)? Because the measures are not independent of each other. Or without some strange analogy - because the hyperparameter set that achieves high accuracy also achieved high recall_micro - and vice versa. $\endgroup$ Apr 26, 2018 at 11:54
  • $\begingroup$ Ok, I was just referring to the scikit learn doc: Note that for “micro”-averaging in a multiclass setting with all labels included will produce equal precision, recall and F, while “weighted” averaging may produce an F-score that is not between precision and recall. Since I want to use a metric already implemented by sci kit learn, which would you recommend for imbalanced multiclass problems? (Especially the averaging method: weighted, micro, macro, ..) $\endgroup$
    – Christian
    Apr 26, 2018 at 11:59
  • $\begingroup$ Weighted - weights should be the class sample numbers $\endgroup$ Apr 26, 2018 at 12:13
  • $\begingroup$ Ok, but my problem is that if I calculate recall_weighted and accuracy for a multiclass problem I'm getting the exact same results (using the sklearn implementations).I'm just interested if there is an actual difference in the multiclass evaluation case. However if I choose recall_macro all classes are equal and I get naturally a lower result, just not sure if this is useful or not. recall_macro should be the metric you meant with "classwise accuracy"? $\endgroup$
    – Christian
    Apr 26, 2018 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.