8
$\begingroup$

In Chapter 3 of the Neural Networks and Deep Learning book, the text repeatedly states that neuron saturation depends only on the activation function of the output layer and the cost function, such as:

"When should we use the cross-entropy instead of the quadratic cost? In fact, the cross-entropy is nearly always the better choice, provided the output neurons are sigmoid neurons."

and,

"This shows that if the output neurons are linear neurons then the quadratic cost will not give rise to any problems with a learning slowdown. In this case the quadratic cost is, in fact, an appropriate cost function to use."

However, it's unclear to me why saturation is only a problem for the output layer. If there are previous hidden layers with sigmoid activations and a quadratic cost function, wouldn't the gradient for those previous layers also have a problem with saturation?

$\endgroup$

3 Answers 3

8
$\begingroup$

It seems to me the author didn't mean that it is the only reason for learning slowdown.

Surely sigmoid activation functions in hidden layers are likely to cause vanishing gradients, but for sigmoid in the output layer, it can be avoided by using the cross-entropy loss.

I think the discussions about output layer and saturation in that chapter is aimed at answering the question when should we use the cross-entropy instead of the quadratic cost? The answer of which is, sigmoid output goes well with cross-entropy loss, and linear output goes well with quadratic loss.

$\endgroup$
2
  • $\begingroup$ Ah, so a cross-entropy loss won't have any effect on vanishing gradients (for sigmoid neurons) before the output layer? $\endgroup$
    – kennysong
    Apr 23, 2016 at 16:23
  • 3
    $\begingroup$ @kennysong right, according to back propagation, once the derivative of the output layer is computed, it won't affect how the derivatives of the hidden layers are computed thereafter. $\endgroup$
    – dontloo
    Apr 24, 2016 at 3:52
0
$\begingroup$

From what I understand, using the cross entropy cost function instead of quadratic cost function will only help you for avoiding the vanishing gradients because of the $\sigma'$ term in the output layer. If we look at the backprop equations, we see that the $\sigma'$ terms are multiplied for gradient computations for every layer except the output layer irrespective of the cost function.

$\endgroup$
0
$\begingroup$

As the problem with quadratic loss is that it learns slowly when it reaches saturation and it does not happen in the case of cross-entropy because of the σ′ term in the formula of the quadratic cost. And as the σ′ term vanishes in the cross-entropy this prevents this slow learning at saturation in the cross-entropy.

Sigmoid functions in hidden layers are likely to cause vanishing gradients, but for sigmoid in the output layer, it can be avoided by using the cross-entropy loss.And this might be the answer for your question!

$\endgroup$
1
  • $\begingroup$ The reason for saturation has nothing to do with the cost function. It is a function of the activation and initialization scheme being used. The loss could be anything but if the activation function is sigmoid and weight initialization is uniform, neurons in the last hidden layer will always saturate (of course if depends on the depth but it has been shown empirically that this happens for networks with 4 or more layers) $\endgroup$
    – kusur
    Apr 15, 2020 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.