1
$\begingroup$

From reading various articles and questions posted on this site, I understand how sometimes layers of a deep neural network may not learn at the same rates, and some (especially earlier) layers may end up having gradients close to zero which supposedly causes the network to learn at a much slower rate.

I find this somewhat counterintuitive from a Mathematical perspective. My confusion is this: gradient descent is used to minimize an objective function and update the values of variables so that the gradient is as close to zero as possible. In this scenario, isn't it inevitable that some of the gradients will be 0? Why is this a problem mathematically? Why do layers with close-to-zero gradients still have much learning to do when they're already at an ideal value? Why can't other layers still be trained while some layers (say earlier layers) have gradients close to 0?

Thanks!

$\endgroup$
1
$\begingroup$

There are two problems going on here. The first is general for very high dimensional optimisation. Think of a one-dimensional minimisation problem, minimising $f(x)$. If you use gradient descent, you take $$x_{k+1}\rightarrow x_k-\epsilon f'(x_k)$$ and the problem is now to choose $\epsilon$. It's not trivial, because even if you take $g(x)=cf(x)$ or $h(x)=f(k\times(x-x_{opt}))$, simply rescaling the input or output variables, good values of $\epsilon$ will change.

One way to choose $\epsilon$ is to use (an approximation to) $f''(x)$, as in Newton-type methods. $$x_{k+1}\rightarrow x_k-\frac{ f'(x_k)}{f''(x_k)}$$ Or you might have some idea from domain knowledge about the correct scaling of the problem.

But in deep neural networks you can't afford the computation to get the second derivative. So, a Newton-type algorithm might be ok if you could afford one, but you can't. And part of the point is avoiding manual feature engineering, so scaling by hand is out.

On top of that, there's a problem with the specific structure of deep neural networks as a feed-forward network of individual nodes. From the chain rule, the derivative with respect to a parameter at layer $i$ involves the product of per-node derivatives at all layers from $i$ onwards. If the derivative at a later layer is near zero, the derivative wrt layer $i$ parameters will also be near zero even if those parameters are far from their optima. The later-layer node may be close to its optimum, but it also may have derivative close to zero because it's far from its optimum and so it isn't sensitive to its inputs. Having a derivative near zero is necessary for being near the optimum of a smooth function, but it is not sufficient.

The classical sigmoid activation function has this problem in both tails: it flattens out for large positive and negative inputs and has exponentially small derivative with increasing distance from the optimum. The arctanh function has it slightly less serously (by a constant factor). The ReLU activation has it much more severely in the lower tail but doesn't have it in the upper tail. The leaky ReLU has the problem less severely.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.