Consider a classification setting like a medical test: Not finding an existing health issue might be much worse (by a factor of 50) than assuming an issue when there is none. I.e. a setting in which false negatives are much worse than false positives or vice versa.

In classification, cross entropy is a very popular loss function with a solid theoretical foundation. However, it does not consider different cost incurred by making a specific type of error.

According to this answer, simply adding a coefficient does not work as intended.

Do you know loss functions considering different severities of error types? I have not found any reference to such a function in my search.

I am aware of the fact that I could use class balancing and sampling techniques. However, I am not excited by the resulting model performance. When I used a different loss function however, the results were much better. The intuition behind the linked loss function is to use the f1-score, which is typically used as a metric as a loss function instead. Because the original metric does not possess the necessary mathematical properties of a loss functions, it is modified in way so that it does. I then modified the linked "f1-loss" to this

Generalized F-score,

in order to be able to penalize false negatives using the parameter beta (when recall is given a higher weight, then the FN-rate will be lower).

Even though the results are much better (compared to class balancing), I would like to use a loss function which is not just a repurposed metric. Ideally, I hope to use a loss function which is proper in the sense that it accounts for the different costs and is theoretically justified.

The equivalent problem in a regression setting might be stated as different costs for over- or underestimating the target variable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.