Why are some loss functions better at optimizing one type of problem than another? In other words, what are some factors to consider when choosing one cost function over another?


The loss function which you want to minimize should mimic the metric you use to evaluate the performance of your model.

The problem is that sometimes your metric does not have a nice derivative which forces you to compromise, because many optimization algorithms require the loss to be nicely differentiable (such as neural networks).

In regression problems, usually $L=\sum_i(y_i-\hat y_i)^2$ (mean square error) is the loss function used, even when the metric is the mean absolute error: $L=\sum_i|y_i-\hat y_i|$, for the reason I explained before.

In classification problems, you would minimize either a cross-entropy function to maximize for example accuracy (which is just $\sum_i y_i=\hat y_i$, i.e. the number of times your prediction is correct).

Notice that sometimes you want to adjust your loss function to whatever metric you use for evaluation. Many Kaggle competition winners do just that. For instance, consider what this recent winner as done to modify the loss to better reflect the F2 metric used for the evaluation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.