Does Cross-Entropy cost affect earlier layers in comparison to MSE cost?

Question

Most of my knowledge about cross-entropy cost is from this tutorial

With cross-entropy the partial derivative to the error with respect to a weight in the output layer is: $$ \dfrac{∂C}{∂w_{k}} = a_k*\dfrac{σ′(z)}{σ(z)(1−σ(z))} * (o_{out} - o_{target}) $$ Which simplifies, according to the guide, to: $$ \dfrac{∂C}{∂w_{k}} = a_k * (o_{out} - o_{target}) $$

Where in MSE it's: $$ \dfrac{∂C}{∂w_{k}} = a_k * a'_{output} * (o_{out} - o_{target}) $$

The removal of $ a'_{output} $ is what should give cross-entropy it's superiority from my understanding, since the partial derivative will be bigger?

But my question is about earlier layers, and if this new cost function affect the training of these layers?

The tutorial did only define a change for the output layer, but not for any amount of hidden layers. So if the calculation of the partial derivative is the same, how does this new cost function affect the training of earlier layers? If so?

And a follow up question would be about a deep neural network with 100 hidden layers. If this new cost function does not affect the training of earlier layers, does it then really do much for a very deep neural network, and its training?

Edit:

When I above use the phrases like "... does this new cost function affect the training of earlier layers?" I mean in comparison to MSE. And more clearly: Will the partial derivative be the same for any hidden layer as if calculated using the MSE cost function?

The output layer is a sigmoid layer.

Answer 1

The removal of $a′_{output}$ is what should give cross-entropy it's superiority from my understanding, since the partial derivative will be bigger?

No. Note that cross entropy's domain is $[0, 1]$, and MSE is calculated as square of arbitrary real number.

In other words, the two error functions have different scales - this should be hopefully reflected in your network's output (cross entropy is typically applied to softmax or sigmoid output).

But my question is about earlier layers, and if this new cost function affect the training of these layers?

If the cost function didn't affect previous layers, then they wouldn't learn anything. But this is exactly the purpose of backpropagation - propagate derivatives so that gradient descent will change previous layers.

It's not clear whether you mean that nothing gets propagated, or you sort of rediscovered vanishing gradient problem yourself.

Answer 2

The removal of $a′_{output}$ is what should give cross-entropy it's superiority from my understanding, since the partial derivative will be bigger?

The partial derivative in MSE will not be bigger here, rather it will be smaller, and that will create a lot of difficulty in learning when the output layer neuron is badly wrong at the beginning of learning - far more difficult than when it's just a little wrong.

For example, if the output layer target was to predict 0, but suppose it predicted 0.9 at the beginning and started learning from there. Now think about the shape of the sigmoid function: it's flatter when it is close to 1, and if you take $a′_{output}$ at the position of 0.9, it will be very small. Eventually, in the derivatives of the cost function, the large error spotted by the $(o_{out}−o_{target})$ term will be suppressed by the small values of $a′_{output}$. So, the gradient $∂C/∂w$ and $∂C/∂b$ will be very small for a long time during the backpropagation and thus the network will learn very slowly to come out from this saturation at the wrong end. However, in the case of cross-entropy since there is no $a′_{output}$ term, the $(o_{out}−o_{target})$ term is able to make the $∂C/∂w$ and $∂C/∂b$ very large when the output neuron is badly wrong, as a result, no neuron saturation will occur here and the learning in the network is very faster.

But my question is about earlier layers, and if this new cost function affect the training of these layers?

The new cost function affects the training of the earlier layers through backpropagation. During the backpropagation, the error in the output layer is propagated in the earlier layer. Then the derivatives in the cost function in the earlier layers will be updated based on the errors in the later layers and the weights and biases will be updated based on these derivatives. [Please go through chapter-2 from the book of Michael Nielsen for detailed understanding of the backpropagation]

And a follow up question would be about a deep neural network with 100 hidden layers. If this new cost function does not affect the training of earlier layers, does it then really do much for a very deep neural network, and its training?

The new cost function will not affect the training of the earlier layers if and only if there is a problem of vanishing gradient or neuron saturation at the wrong end, and if that occurs, that would badly affect the training process.

Answer 3

I'm studying the same tutorial and having the same question too.

From my perspective, you might want to ask whether the cancellation of $\sigma'(z^l)$ works for the earlier layers too.

The answer is No.

In chapter 5, Michael Nielsen described the vanishing gradient problem. Although the cross-entropy was used, vanishing gradient problem still happened for the sigmoid function. Michael Nielsen explained why here. Please check the equation (122) which was the partial derivative for the bias in the first layer, the cancellation only worked for the output layer in the $\sigma'(z_4) \, \frac{\partial C}{\partial a_4}$, the rest part $\sigma'(z_1) \, w_2 \sigma'(z_2) \, w_3 \sigma'(z_3) \, w_4$ contained three $\sigma'(z^l)$s, which resulted in the vanishing gradient problem.

$$\begin{eqnarray} \frac{\partial C}{\partial b_1} = \sigma'(z_1) \, w_2 \sigma'(z_2) \, w_3 \sigma'(z_3) \, w_4 \sigma'(z_4) \, \frac{\partial C}{\partial a_4}. \tag{122}\end{eqnarray}$$

Acutally, the vanishing gradient problem caused by $\sigma'(z^l)$ is easier to learn in the equations of BP (chapter 2 and copied below). The cancellation of $\sigma'(z^l)$ only works in equation (BP1), and $\sigma'(z^l)$ still exists in equation (BP2), which results in the vanishing gradient problem.

Maybe the ReLU activation solve the problem, I don't know, I'm a beginner of AI.