I have just read about different cost functions for training neural networks. How to determine which cost function is applicable in any given situation?

  • $\begingroup$ If you give examples of particular cost functions that you have in mind, I can expand on my answer. At the moment the question is rather vague and my answer thus rather general. $\endgroup$ – Jan Kukacka May 8 at 10:22

Cost function is an essential part of neural network training. It tells the model if and how much are its predictions wrong, and in which direction should they change to be better. The second part implies that the cost function needs to be differentiable w.r.t. model predictions. Since the question is not very specific, I will focus on supervised learning types of loss functions in my answer.

There are two basic cost functions we use all the time:

1. Mean squared error

Defined as $\mathcal L(w) = \frac1N \sum_n^N \lVert f_w(x_n) - y_n \rVert_2^2 $, this loss function is used for regression problems with continuous targets $y$.

2. Multinomial cross-entropy

Defined as $\mathcal L(w) = - \frac1N \sum_n^N\sum_k^K y_{n,k}\log f_{w,k}(x_{n})$, this loss function is used for classification problems with discrete targets.

If you are not sure, use one the simple rule:

  • Regression $\rightarrow$ MSE
  • Classification $\rightarrow$ Cross-entropy

In some cases, however, it is beneficial to use a loss function that better reflects the properties of the problem you are trying to solve. Generally, as a form of regularization, it is beneficial to include as much domain knowledge as you have into the model and learning.

For example, the MSE loss assumes normally distributed errors on the data, which is not necessarily always the case. Multinomial cross entropy struggles with class imbalance, in which case using its weighted version may be the better choice. There are other loss functions designed for specific types of problems:

  1. Dice loss for image segmentation [1,3]
  2. Combination of smooth L1 and cross-entropy losses for object detection [2]
  3. there's much more...


[1]: Sudre et al., 2017: Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations

[2]: Girschick, 2015: Fast R-CNN

[3]: Q: Dice-coefficient loss function vs cross-entropy


Not the answer you're looking for? Browse other questions tagged or ask your own question.