What are the applications of different cost functions and which one to choose? I have just read about different cost functions for training neural networks. How to determine which cost function is applicable in any given situation?
 A: Cost function is an essential part of neural network training. It tells the model if and how much are its predictions wrong, and in which direction should they change to be better. The second part implies that the cost function needs to be differentiable w.r.t. model predictions. Since the question is not very specific, I will focus on supervised learning types of loss functions in my answer.
There are two basic cost functions we use all the time:
1. Mean squared error
Defined as $\mathcal L(w) = \frac1N \sum_n^N \lVert f_w(x_n) - y_n \rVert_2^2 $, this loss function is used for regression problems with continuous targets $y$.
2. Multinomial cross-entropy
Defined as $\mathcal L(w) = - \frac1N \sum_n^N\sum_k^K y_{n,k}\log f_{w,k}(x_{n})$, this loss function is used for classification problems with discrete targets.
If you are not sure, use one the simple rule:


*

*Regression $\rightarrow$ MSE

*Classification $\rightarrow$ Cross-entropy



In some cases, however, it is beneficial to use a loss function that better reflects the properties of the problem you are trying to solve. Generally, as a form of regularization, it is beneficial to include as much domain knowledge as you have into the model and learning.
For example, the MSE loss assumes normally distributed errors on the data, which is not necessarily always the case. Multinomial cross entropy struggles with class imbalance, in which case using its weighted version may be the better choice. There are other loss functions designed for specific types of problems:


*

*Dice loss for image segmentation [1,3]

*Combination of smooth L1 and cross-entropy losses for object detection [2]

*there's much more...

References
[1]: Sudre et al., 2017: Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
[2]: Girschick, 2015: Fast R-CNN
[3]: Q: Dice-coefficient loss function vs cross-entropy
