4
$\begingroup$

I am unclear about max a posteriori and squared loss.

Let me assume I have $N$ images and $\mathbf{y}_i$ is the label of the image $i$, where, $\mathbf{y}_i\in \mathbb{R}^{C\times 1}$ - a binary vector of lenght $C$ (number of classes). $\mathbf{y}_{ic} = 1$ when the image $i$ belonging to class $c$.

Consider the following two loss functions.

\begin{align} L_1 &= -\sum_i\sum_{c=1}^C y_{ic}\, \log(P(y_{ic}|\mathcal{D})) \\ L_2 &= \sum_i \sum_{c=1}^C \left(y_{ic}-P(y_{ic}|\mathcal{D})\right)^2 \end{align}

What is the difference between these functions? When should I use $L_1$ and when should I use $L_2$?

I observed that in neural networks always $L_2$ is used. Why?

Improve this question
$\endgroup$
1
  • $\begingroup$ That is not true. $L_1$ is used more which is the cross entropy loss. $\endgroup$
    – n0p
    Jul 7, 2016 at 17:13

1 Answer 1

Reset to default
4
$\begingroup$

L2 is not always used with neural networks, indeed for statistical pattern recognition problems the cross-entropy loss (with a softmax activation function for the output layer) is the preferred option. See chapter 6 of Bishop's excellent book on Neural Networks for Pattern Recognition.

Secondly, I would say that L1 would be a maximum likelihood approach, rather than maximum a-posteriori as there is no prior distribution involved, just a likelihood.

Asymptotically (in the limit of an infinite amount of data and hidden units), both L1 and L2 will give the same answer, because the minimiser of L1 is given when the output of the model is the true probability of class membership, and the minimiser of L2 is the conditional mean of the target variable, which in this case is also the true probability of class membership.

The difference arises away from these asymptotic conditions (i.e. more or less every practical case), where I would suggest that L1 is probably more efficient in converging to the optimal solution in terms of the number of training samples given, but I am not confident that the practical difference is likely to be great for most problems.

In practice, most reasonable loss functions are suitable for training neural networks (see Saerens et al), however I almost always use a maximum likelihood criterion for training neural networks because this is the most theoretically sound approach as it most accurately describes the variability of the target variable around its conditional mean.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the detailed response. P(y|D) is it likelihood or posterier? Could you please comment on this? $\endgroup$
    – user570593
    Jan 7, 2015 at 14:25
  • $\begingroup$ I meant that L1 was a maximum likelihood loss function as it is being used to optimise the parameters of the model that give p(y|D) by minimising the negative log likelihood of the model given the data. I would probably write p(y|w) rather the p(y|D), where w is the weights of the model. $\endgroup$
    – Dikran Marsupial
    Jan 7, 2015 at 14:39
  • $\begingroup$ I thought p(W|y) is the likelihood. P(y|W) is the posterior. Am I wrong? $\endgroup$
    – user570593
    Jan 7, 2015 at 15:02
  • $\begingroup$ I think the problem is that posterior can be used in at least two ways here: p(y|x;w) can mean the posterior probability of the class label after having seen the corresponding vector of inputs, x. However, p(w|D) $\propto$ p(D|w)p(w) would be the posterior distribution of the weights given the data, where p(w|D) is the likelihood, which is essentially L1. Hence minimizing L1 is maximum likelihood, rather than maximum a-posteriori estimation as there is no p(w). en.wikipedia.org/wiki/Maximum_a_posteriori_estimation $\endgroup$
    – Dikran Marsupial
    Jan 7, 2015 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.