21
$\begingroup$

I have been reading a lot about convoloutional neural networks and was wondering how they avoid the vanishing gradient problem. I know deep belief networks stack single level auto-encoders or other pre-trained shallow networks and can thus avoid this problem but I don't know how it is avoided in CNNs.

According to Wikipedia:

"despite the above-mentioned "vanishing gradient problem," the superior processing power of GPUs makes plain back-propagation feasible for deep feedforward neural networks with many layers."

I don't understand why GPU processing would remove this problem?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Did the wikipedia article not justify why GPU help to address the vanishing gradient problem? Is it because even though the gradients are small, since GPUs are so fast we still manage to improve the parameters by doing lots of steps thanks to the GPUs? $\endgroup$
    – Charlie Parker
    Commented Aug 24, 2016 at 1:12
  • 2
    $\begingroup$ Exactly. Vanishing gradient problem is the reason why lower layer weights are updated at a very small rate, and thus it takes forever to train the network. But, as with GPUs you can do more computations (i.e. more updates to the weights) in lesser time, with more and more GPU processing, vanishing gradient problem is somewhat vanished to some extent. $\endgroup$
    – exAres
    Commented Nov 30, 2016 at 8:23
  • $\begingroup$ @CharlieParker, could you elaborate on GPU's are fast correlated with vanishing gradients, I can understand the fast logic with large memory bandwidth to process multiple matrix multiplications! but could you please explain what it has to do with the derivatives? The vanishing gradient issue seems to do more with weight initialization, isn't it! $\endgroup$
    – Anu
    Commented Feb 5, 2019 at 19:33
  • $\begingroup$ Because vanishing gradient is not really a problem, it is a feature of Buttefly effect. A derivative of a derivative of a derivative of a derivative, all those are multiplied and the multiplication by fraction get another smaller fraction. GPUs are just faster to train, but if you use CPU you would achieve the same results as GPU if you train it for a few months. $\endgroup$
    – Nulik
    Commented Apr 7, 2020 at 17:24

1 Answer 1

Reset to default
17
$\begingroup$

The vanishing gradient problem requires us to use small learning rates with gradient descent which then needs many small steps to converge. This is a problem if you have a slow computer which takes a long time for each step. If you have a fast GPU which can perform many more steps in a day, this is less of a problem.

There are several ways to tackle the vanishing gradient problem. I would guess that the largest effect for CNNs came from switching from sigmoid nonlinear units to rectified linear units. If you consider a simple neural network whose error $E$ depends on weight $w_{ij}$ only through $y_j$, where

$$y_j = f\left( \sum_iw_{ij}x_i \right),$$

its gradient is

\begin{align} \frac{\partial}{\partial w_{ij}} E &= \frac{\partial E}{\partial y_j} \cdot \frac{\partial y_j}{\partial w_{ij}} \\ &= \frac{\partial E}{\partial y_j} \cdot f'\left(\sum_i w_{ij} x_i\right) x_i. \end{align}

If $f$ is the logistic sigmoid function, $f'$ will be close to zero for large inputs as well as small inputs. If $f$ is a rectified linear unit,

\begin{align} f(u) = \max\left(0, u\right), \end{align} the derivative is zero only for negative inputs and 1 for positive inputs. Another important contribution comes from properly initializing the weights. This paper looks like a good source for understanding the challenges in more details (although I haven't read it yet):

http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf

Improve this answer
$\endgroup$
8
  • 2
    $\begingroup$ I'm a little puzzled about the rectified linear units. Yes, for sigmoids etc. the gradient is often very small - but for rectified linear units it is often exactly zero. Isn't that worse? Thus, if the weights of an unit are unfortunate, they will never ever change. $\endgroup$
    – Dr. Hans-Peter Störr
    Commented Sep 15, 2015 at 19:55
  • 2
    $\begingroup$ Thinking about this, leaky and/or noisy ReLUs might be in use for that reason. $\endgroup$
    – sunside
    Commented Aug 16, 2016 at 21:26
  • 8
    $\begingroup$ Why is your first sentence true? I.e. "The vanishing gradient problem requires us to use small learning rates with gradient descent which then needs many small steps to converge." Why do we need small learning rates to deal with the vanishing gradient problem? If the gradients are already small with due to vanishing gradients I would have expected that making them small only made things worse. $\endgroup$
    – Charlie Parker
    Commented Aug 24, 2016 at 1:11
  • 2
    $\begingroup$ Good question, I should have explained that statement better. The vanishing gradient problem is not that all gradients are small (which we could easily fix by using large learning rates), but that the gradients vanish as you backpropagate through the network. I.e., the gradients are small in some layers but large in other layers. If you use large learning rates, the whole thing explodes (because some gradients are large), so you have to use a small learning rate. Using multiple learning rates is another approach to addressing the problem, at the cost of introducing more hyperparameters. $\endgroup$
    – Lucas
    Commented Aug 24, 2016 at 15:16
  • 6
    $\begingroup$ I would argue that the learning rate is mostly tied to the exploding gradient problem. Scaling the gradient down with an exaggeratingly low learning rate does not at all prevent vanishing gradients, it just delays the effect as learning slows down considerably. The effect itself is caused by the repeated application of nonlinearities and multiplication of small values. Of course there is a trend to go to smaller learning rates (due to computing power) but that has nothing to do with vanishing gradients as it only controls how well the state space is explored (given stable conditions). $\endgroup$
    – runDOSrun
    Commented May 4, 2017 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.