Cross-Entropy or Log Likelihood in Output layer

Question

I read this page: http://neuralnetworksanddeeplearning.com/chap3.html

and it said that sigmoid output layer with cross-entropy is quite similiar with softmax output layer with log-likelihood.

what happen if I use sigmoid with log-likelihood or softmax with cross entropy in the output layer? is it fine? becuase I see there's only little difference in equation between cross entropy (eq.57):

$$C = -\frac{1}{n} \sum\limits_x (y \ln a + (1-y) \ln (1-a))$$

and log likelihood (eq.80):

$$C =-\frac{1}{n} \sum\limits_x(\ln a^L_y)$$

Answer 1

The negative log likelihood (eq.80) is also known as the multiclass cross-entropy (ref: Pattern Recognition and Machine Learning Section 4.3.4), as they are in fact two different interpretations of the same formula.

eq.57 is the negative log likelihood of the Bernoulli distribution, whereas eq.80 is the negative log likelihood of the multinomial distribution with one observation (a multiclass version of Bernoulli).

For binary classification problems, the softmax function outputs two values (between 0 and 1 and sum to 1) to give the prediction of each class. While the sigmoid function outputs one value (between 0 and 1) to give the prediction of one class (so the other class is 1-p).

So eq.80 can't be directly applied to the sigmoid output, though it is essentially the same loss as eq.57.

Also see this answer.

---

Following is a simple illustration of the connection between (sigmoid + binary cross-entropy) and (softmax + multiclass cross-entropy) for binary classification problems.

Say we take $0.5$ as the split point of the two categories, for sigmoid output it follows,

$$\sigma(wx+b)=0.5$$ $$wx+b=0$$ which is the decision boundary in the feature space.

For softmax output it follows $$\frac{e^{w_1x+b_1}}{e^{w_1x+b_1}+e^{w_2x+b_2}}=0.5$$ $$e^{w_1x+b_1}=e^{w_2x+b_2}$$ $$w_1x+b_1=w_2x+b_2$$ $$(w_1-w_2)x+(b_1-b_2)=0$$ so it remains the same model although there are twice as many parameters.

The followings show the decision boundaries obtained using theses two methods, which are almost identical.

Answer 2

Expanding on @dontloo's answer, consider a classification task with $K$ classes.

Let's separately look at the output layer of a network and the cost function. For our purpose here, the output layer is either sigmoid or softmax and the cost function is either cross-entropy or log-likelihood.

Output Layers

In the case of a sigmoid, the output layer will have $K$ sigmoids each ouputting a value between 0 and 1. Crucially, the sum of these outputs may not equal one and hence they cannot be interpreted as a probability distribution. The only exception to both statements is the case when $K=2$ i.e. binary classification, when only one sigmoid is sufficient. And, in this case, a second output can be imagined to be one minus the lone output.

If the output layer is softmax, it also has $K$ outputs. But in this case, the outputs sum to one. Because of this constraint, a network with a softmax output layer has less flexibility than one with multiple sigmoids (except, of course, for $K=2$).

To illustrate the constraint, consider a network used to classify digits. It has ten output nodes. If they are sigmoids then two of them, say the ones for 8 and 9 or 0 and 6, can both output, say, 0.9. In the case of softmax, this is not possible. Outputs could still be equal--both 0.45, for example--but because of the constraint, when the weights get adjusted to increase the output for one digit, it necessarily decreases the output for some other digit(s). The text has a slider demo in the same chapter to illustrate this effect.

What about inference? Well, one straightforward method would be to simply assign the class which has the largest output. This is true for both types of output layers.

As for the cost function, it is possible to use either cross-entropy or log-likelihood (or some other cost-function such as mean-squared error) for either networks.

Let's look at that below.

Cost Functions

The cross-entropy cost of a $K$-class network would be $$ C_\text{CE} = -\frac{1}{n} \sum\limits_x \sum\limits_{k=1}^K (y_k \ln a_k^L + (1 - y_k) \ln (1 - a_k^L)) $$ where $x$ is an input and $n$ is the number of examples in the input set. This is equation (63) in the book.

Note that, for each $x$, only one of the $y_k$ is 1 and the rest are 0 (i.e. one-hot encoding).

The log-likelihood cost of a $K$-class network is $$ C_\text{LL} = -\frac{1}{n} \sum\limits_x y^T \ln(a^L) = -\frac{1}{n} \sum\limits_x \sum\limits_{k=1}^K y_k \ln(a_k^L) $$ where $y$ is the (one-hot encoded) desired output and $a^L$ is the output of the model. This is just a different way of writing equation (80) in the book.

Again, note that, for each $x$, only one element of $y$ is 1 and the rest are 0.

Here is the crucial difference between the two cost functions: the log-likelihood considers only the output for the corresponding class, whereas the cross-entropy function also considers the other outputs as well. You can see this in the above expressions--in the summation, both $C_\text{CE}$ and $C_\text{LL}$ have the same first term, but $C_\text{CE}$ has an additional term. What this means is that both CE and LL reward the network for the amount of output in the correct class. But CE also penalizes the network for the amounts in the other classes. If the confusion is strong then the penalty is also strong. Let's illustrate below.

Example

Let's look at a single sample i.e. $n=1$.

Let $$ \begin{align} a^L & = [0.55, 0.02, 0.01, 0.03, 0.01, 0.05, 0.17, 0.01, 0.06, 0.09], \text{and} \\ y & = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0] \end{align} $$ i.e. the input is a 0 and the network has some confusion over a 6, but it's not too bad. This output is applicable both to sigmoid and softmax output layers.

The costs are $$ \begin{align} C_\text{CE} & = 1.0725 \\ C_\text{LL} & = 0.5978 \end{align} $$

Now, let's say look at another scenario where the output now indicates more confusion between a 0 and a 6. We keep the output of the 0-class the same and increase the output of the 6-class and decrease the output of the other classes. Again, this output is applicable to both sigmoid and softmax. $$ a^L = [0.55, 0.002, 0.001, 0.003, 0.001, 0.04, 0.37, 0.001, 0.012, 0.02] $$

What happens to the cost? $$ \begin{align} C_\text{CE} & = 1.1410 \\ C_\text{LL} & = 0.5978 \end{align} $$ As can be seen, the LL cost has not changed. But the CE cost has increased--it has penalized the stronger confusion of a 0 to a 6!

Summary

In summary, yes, the output layers and cost functions can be mixed and matched. They affect how the network behaves and how the results are to be interpreted.

Answer 3

I think that @user650654 made a mistake in his formulation of the Cross Entropy and therefore his conclusion is incorrect. In the case of hard labels (i.e., using one-hot vectors for ground truth, where only one element of the vector is assigned 1 and all others are assigned 0 probability), the Cross Entropy loss and the log-likelihood are equivalent.

Let's assume that $y$ and $\hat{y}$ are the one-hot ground-truth vector and the estimated probability vector that produced by applying the Softmax function. Here I note $y[i]$ as the i-th entry of the $y$ vector. Recall that the definition of Cross Entropy is:

$$\text{CE}(y,\hat{y}) = -\Sigma_{i}y[i]\log(\hat{y}[i])$$

But since $y$ is a one-hot vector, only one of the elements in that sum affects the output, which is simply: $-\log(\hat{y}[i])$, and that is the negative log-likelihood.