Cross-Entropy or Log Likelihood in Output layer I read this page:
http://neuralnetworksanddeeplearning.com/chap3.html
and it said that sigmoid output layer with cross-entropy is quite similiar with softmax output layer with log-likelihood.
what happen if I use sigmoid with log-likelihood or softmax with cross entropy in the output layer? is it fine? becuase I see there's only little difference in equation between cross entropy (eq.57):
$$C = -\frac{1}{n} \sum\limits_x (y \ln a + (1-y) \ln (1-a))$$
and log likelihood (eq.80):
$$C =-\frac{1}{n} \sum\limits_x(\ln a^L_y)$$
 A: The negative log likelihood (eq.80) is also known as the multiclass cross-entropy (ref: Pattern Recognition and Machine Learning Section 4.3.4), as they are in fact two different interpretations of the same formula.
eq.57 is the negative log likelihood of the Bernoulli distribution, whereas eq.80 is the negative log likelihood of the multinomial distribution with one observation (a multiclass version of Bernoulli). 
For binary classification problems, the softmax function outputs two values (between 0 and 1 and sum to 1) to give the prediction of each class. While the sigmoid function outputs one value (between 0 and 1) to give the prediction of one class (so the other class is 1-p).  
So eq.80 can't be directly applied to the sigmoid output, though it is essentially the same loss as eq.57.
Also see this answer.

Following is a simple illustration of the connection between (sigmoid + binary cross-entropy) and (softmax + multiclass cross-entropy) for binary classification problems. 
Say we take $0.5$ as the split point of the two categories, for sigmoid
output it follows,
$$\sigma(wx+b)=0.5$$
$$wx+b=0$$
which is the decision boundary in the feature space.
For softmax output it follows
$$\frac{e^{w_1x+b_1}}{e^{w_1x+b_1}+e^{w_2x+b_2}}=0.5$$
$$e^{w_1x+b_1}=e^{w_2x+b_2}$$
$$w_1x+b_1=w_2x+b_2$$
$$(w_1-w_2)x+(b_1-b_2)=0$$
so it remains the same model although there are twice as many parameters.
The followings show the decision boundaries obtained using theses two methods, which are almost identical.


A: Expanding on @dontloo's answer,
consider a classification task with $K$ classes.
Let's separately look at the output layer of a network and the cost
function.  For our purpose here, the output layer is either sigmoid or
softmax and the cost function is either cross-entropy or log-likelihood.
Output Layers
In the case of a sigmoid, the output layer will have $K$ sigmoids each
ouputting a value between 0 and 1.  Crucially, the sum of these
outputs may not equal one and hence they cannot be interpreted as a
probability distribution.  The only exception to both statements is
the case when $K=2$ i.e. binary classification, when only one sigmoid
is sufficient.  And, in this case, a second output can be imagined to
be one minus the lone output.
If the output layer is softmax, it also has $K$ outputs.  But in this
case, the outputs sum to one.  Because of this constraint, a network
with a softmax output layer has less flexibility than one with multiple sigmoids (except, of course, for $K=2$).
To illustrate the constraint, consider a network used to classify
digits.  It has ten output nodes.  If they are sigmoids then two of
them, say the ones for 8 and 9 or 0 and 6, can both output, say, 0.9.
In the case of softmax, this is not possible.  Outputs could still be
equal--both 0.45, for example--but because of the constraint, when the
weights get adjusted to increase the output for one digit, it necessarily
decreases the output for some other digit(s).  The text has a slider
demo in the same chapter to illustrate this effect.
What about inference?  Well, one straightforward method would be to
simply assign the class which has the largest output.  This is true
for both types of output layers.
As for the cost function, it is possible to use either cross-entropy
or log-likelihood (or some other cost-function such as mean-squared
error) for either networks.
Let's look at that below.
Cost Functions
The cross-entropy cost of a $K$-class network would be
$$
C_\text{CE} = -\frac{1}{n} \sum\limits_x \sum\limits_{k=1}^K (y_k \ln a_k^L + (1 - y_k) \ln (1 - a_k^L))
$$
where $x$ is an input and $n$ is the number of examples in the input
set.  This is equation (63) in the book.
Note that, for each $x$, only one of the $y_k$ is 1 and the rest are 0
(i.e. one-hot encoding).
The log-likelihood cost of a $K$-class network is
$$
C_\text{LL} = -\frac{1}{n} \sum\limits_x y^T \ln(a^L) = -\frac{1}{n} \sum\limits_x \sum\limits_{k=1}^K y_k \ln(a_k^L)
$$
where $y$ is the (one-hot encoded) desired output and $a^L$ is the output of
the model.  This is just a different way of writing equation (80) in the book.
Again, note that, for each $x$, only one element of $y$ is 1 and the rest are 0.
Here is the crucial difference between the two cost functions: the
log-likelihood considers only the output for the corresponding class,
whereas the cross-entropy function also considers the other outputs as
well.  You can see this in the above expressions--in the summation,
both $C_\text{CE}$ and $C_\text{LL}$ have the same first term, but
$C_\text{CE}$ has an additional term.  What this means is that both CE
and LL reward the network for the amount of output in the correct
class.  But CE also penalizes the network for the amounts in the other
classes.  If the confusion is strong then the penalty is also strong.
Let's illustrate below.
Example
Let's look at a single sample i.e. $n=1$.
Let
$$
\begin{align}
a^L & = [0.55, 0.02, 0.01, 0.03, 0.01, 0.05, 0.17, 0.01, 0.06, 0.09], \text{and} \\
y & = [1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
\end{align}
$$
i.e. the input is a 0 and the network has some
confusion over a 6, but it's not too bad.  This output is applicable
both to sigmoid and softmax output layers.
The costs are
$$
\begin{align}
C_\text{CE} & = 1.0725 \\
C_\text{LL} & = 0.5978
\end{align}
$$
Now, let's say look at another scenario where the output now indicates
more confusion between a 0 and a 6.  We keep the output of the 0-class
the same and increase the output of the 6-class and decrease the
output of the other classes.  Again, this output is applicable to both
sigmoid and softmax.
$$
a^L = [0.55, 0.002, 0.001, 0.003, 0.001, 0.04, 0.37, 0.001, 0.012, 0.02]
$$
What happens to the cost?
$$
\begin{align}
C_\text{CE} & = 1.1410 \\
C_\text{LL} & = 0.5978
\end{align}
$$
As can be seen, the LL cost has not changed.  But the CE cost has
increased--it has penalized the stronger confusion of a 0 to a 6!
Summary
In summary, yes, the output layers and cost functions can be mixed and
matched.  They affect how the network behaves and how the results are
to be interpreted.
A: I think that @user650654 made a mistake in his formulation of the Cross Entropy and therefore his conclusion is incorrect. In the case of hard labels (i.e., using one-hot vectors for ground truth, where only one element of the vector is assigned 1 and all others are assigned 0 probability), the Cross Entropy loss and the log-likelihood are equivalent.
Let's assume that $y$ and $\hat{y}$ are the one-hot ground-truth vector and the estimated probability vector that produced by applying the Softmax function. Here I note $y[i]$ as the i-th entry of the $y$ vector. Recall that the definition of Cross Entropy is:
$$\text{CE}(y,\hat{y}) = -\Sigma_{i}y[i]\log(\hat{y}[i])$$
But since $y$ is a one-hot vector, only one of the elements in that sum affects the output, which is simply: $-\log(\hat{y}[i])$, and that is the negative log-likelihood.
