Consider a classification setting like a medical test: Not finding an existing health issue might be much worse (by a factor of 50) than assuming an issue when there is none. I.e. a setting in which false negatives are much worse than false positives or vice versa.
In classification, cross entropy is a very popular loss function with a solid theoretical foundation. However, it does not consider different cost incurred by making a specific type of error.
According to this answer, simply adding a coefficient does not work as intended.
Do you know loss functions considering different severities of error types? I have not found any reference to such a function in my search.
I am aware of the fact that I could use class balancing and sampling techniques. However, I am not excited by the resulting model performance. When I used a different loss function however, the results were much better. The intuition behind the linked loss function is to use the f1-score, which is typically used as a metric as a loss function instead. Because the original metric does not possess the necessary mathematical properties of a loss functions, it is modified in way so that it does. I then modified the linked "f1-loss" to this
,
in order to be able to penalize false negatives using the parameter beta (when recall is given a higher weight, then the FN-rate will be lower).
Even though the results are much better (compared to class balancing), I would like to use a loss function which is not just a repurposed metric. Ideally, I hope to use a loss function which is proper in the sense that it accounts for the different costs and is theoretically justified.
The equivalent problem in a regression setting might be stated as different costs for over- or underestimating the target variable.
