2
$\begingroup$

Let's consider a multi-class classification problem with 4 classes: 0, 1, 2, and 3

F1-Score 'macro'-averaged is calculated like that:

F1_macro = 0.25*F1_class0 + 0.25*F1_class1 + 0.25*F1_class2 + 0.25*F1_class3

On the other hand, supposing that the relative supports for the classes are:

  • class 0: 0.4
  • class 1: 0.1
  • class 2: 0.4
  • class 3: 0.1

F1-Score 'weighted'-averaged is equivalent to:

F1_weighted = 0.4*F1_class0 + 0.1*F1_class1 + 0.4*F1_class2 + 0.1*F1_class3

Now my question is: does it make sense to consider a customized version of the scores where the weights' values follow an empirical rule? Let's say for example that I am interested in obtaining a model that predicts well minority classes, and in particular class 3. Could the following metric be a valid approach?

F1_custom = 0.1*F1_class0 + 0.35*F1_class1 + 0.1*F1_class2 + 0.45*F1_class3

I understand that considering the weights as inversely proportional to the relative support could result in overproportional weights in case of severe imbalance, but considering them like that should not be a problem in this sense.

I am aware that there are better metrics, that are also threshold-independent (e.g. AUPRC or AP) that can be extended for a multi-class problem and for which I could also apply this method, but for the moment I am interested in validating this 'custom averaging' method, so I would like to understand if it is feasible for a simple metric like F1-Score and then extend it to other metrics (like AP).

Am I missing something? Why I can not find anything about this topic on internet?

$\endgroup$

1 Answer 1

1
$\begingroup$

In principle, there is no problem with defining your custom scores. This can be a custom f1-score that you care about, a customized score obtained from confusion matrix, or some other arbitrary formula. The important thing is, it should make sense regarding your problem.

Let's say for example that I am interested in obtaining a model that predicts well minority classes, and in particular class 3. Could the following metric be a valid approach?

F1_custom = 0.1*F1_class0 + 0.45*F1_class1 + 0.1*F1_class2 + 0.35*F1_class3

Regarding this, I'd expect you give more weight to the F1 score of class 3.

Am I missing something? Why I can not find anything about this topic on internet?

Maybe you can find more people doing customizations by looking for custom losses defined on confusion matrices. People usually assign a cost of misclassification for each class pair, i.e. $C_{ij}$, and multiply the costs with the entries in the confusion matrix.

$\endgroup$
1
  • $\begingroup$ Thank you so much for your answer! There was a typo in the text, I fixed it, than you for noticing it, you can edit your answer as well :) $\endgroup$ May 8, 2023 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.