22
$\begingroup$

I read this page: http://neuralnetworksanddeeplearning.com/chap3.html

and it said that sigmoid output layer with cross-entropy is quite similiar with softmax output layer with log-likelihood.

what happen if I use sigmoid with log-likelihood or softmax with cross entropy in the output layer? is it fine? becuase I see there's only little difference in equation between cross entropy (eq.57):

$$C = -\frac{1}{n} \sum\limits_x (y \ln a + (1-y) \ln (1-a))$$

and log likelihood (eq.80):

$$C =-\frac{1}{n} \sum\limits_x(\ln a^L_y)$$

$\endgroup$
  • $\begingroup$ the link does not seem to work $\endgroup$ – dontloo Feb 23 '16 at 5:44
  • $\begingroup$ ah, sorry, edited $\endgroup$ – malioboro Feb 23 '16 at 5:45
42
$\begingroup$

The negative log likelihood (eq.80) is also known as the multiclass cross-entropy (ref: Pattern Recognition and Machine Learning Section 4.3.4), as they are in fact two different interpretations of the same formula.

eq.57 is the negative log likelihood of the Bernoulli distribution, whereas eq.80 is the negative log likelihood of the multinomial distribution with one observation (a multiclass version of Bernoulli).

For binary classification problems, the softmax function outputs two values (between 0 and 1 and sum to 1) to give the prediction of each class. While the sigmoid function outputs one value (between 0 and 1) to give the prediction of one class (so the other class is 1-p).

So eq.80 can't be directly applied to the sigmoid output, though it is essentially the same loss as eq.57.

Also see this answer.


Following is a simple illustration of the connection between (sigmoid + binary cross-entropy) and (softmax + multiclass cross-entropy) for binary classification problems.

Say we take $0.5$ as the split point of the two categories, for sigmoid output it follows,

$$\sigma(wx+b)=0.5$$ $$wx+b=0$$ which is the decision boundary in the feature space.

For softmax output it follows $$\frac{e^{w_1x+b_1}}{e^{w_1x+b_1}+e^{w_2x+b_2}}=0.5$$ $$e^{w_1x+b_1}=e^{w_2x+b_2}$$ $$w_1x+b_1=w_2x+b_2$$ $$(w_1-w_2)x+(b_1-b_2)=0$$ so it remains the same model although there are twice as many parameters.

The followings show the decision boundaries obtained using theses two methods, which are almost identical.

$\endgroup$
  • 1
    $\begingroup$ hmm .. I understand.. thanks for your answer :) $\endgroup$ – malioboro Feb 23 '16 at 13:27
  • 1
    $\begingroup$ @malioboro you're welcome :) $\endgroup$ – dontloo Feb 24 '16 at 1:39

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.