I wanted to train a network with non-linearities that suffer from the vanishing (or exploding gradient problem though mainly vanishing). I know that the (current) standard way is to use batch normalization 1 [BN]1 or simply abandon the non-linearity and use ReLu Rectifier/ReLu units.

I wanted two things:

  1. Stick with my non-linearity, so I don't want to abandon it and use the ReLu (i.e. no ReLu's allowed!). Re-parametrising the non-linearity is ok, say putting a multiplicative in front of it as in $\theta(s)$ for example.
  2. Ideally, I did not want to rely too much batch normalization (or at least if its used, it has to be used in a novel way other than how it was used in the original paper or generalize to many non-linearities). One of the reasons I wanted to avoid Batch Normalize is because it seems to only work for specific non-linearities. For example, for sigmoids, tanh but its unclear how they'd work for other non-linearities, say gaussians.

The reason I have these constraints is because I'd like to deal with the problem of vanishing gradient or exploding gradients by taling the problem directly rather than hacking a solution that works only for specific non-linearities or just avoiding the problem by shoving in a ReLu.

I was wondering, with those two constraints, what are alternative ways to deal with the vanishing gradient problem? (another non-linearity in consideration would be RBF gaussian kernel with euclidean norm pre-activation, sigmoid, tanh, etc)

The possible (vague) ideas I had in mind would be:

  1. Have good initialization so that the saturating non-linearities don't start already saturated (saturated non-linearities result in gradients close to zero).
  2. For RBF, similarly, good init might be important because gaussians mostly have a large value close to 0 (i.e. when filters are similar to its activation or data). Thus, having them too big or too small has a similar vanishing gradient issues.
  3. I don't really know if this is too constraining but it would be nice if there was a different way to use batch normalization other than its traditional suggestion in the original paper (or maybe some BN idea that generalizes to a bigger set of non-linearities, currently it seems most of the research is to show it works for sigmoids as far as I know).
  4. Another idea could be to instead of having non-linearity $\theta(z)$ we have $a \theta(z) $ where $a \in \mathbb{R}$. If $a > 1$, then it means that the non-linearities are not multiplied backwards multiple times for each layer, so as to avoid to be "vanished" for earlier layers. It might make the learning rule unstable, so maybe some regularizer might be a good idea.
  5. An optimizer that intrinsically deals with the vanishing gradient (or at least updating each parameter differently). For example, if its a layer closer to the input, then the learning step should be larger. It would be nice for the learning algorithm to take this into account by itself so to deal with the vanishing gradient.

If there are any suggestions on how to deal with vanishing gradient other than batch-norm or ReLu's I'd love to hear about them!

It seems that vanishing gradient happens mainly because the non-linearities have the property that $ |a| < 1$ and also because $ | \theta'(s) | < 1$ and after multiplying it many times, it either explodes or vanish. Explicitly saying the problem might help solve it. The issue is that it causes lower layers to not update or hinders signal through the network. It would be nice to maintain this signal flowing through the network, during the forward and backward pass (and also during training, not only at initialization).

1: Ioffe S. and Szegedy C. (2015),
"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift",
Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
Journal of Machine Learning Research: W&CP volume 37

  • $\begingroup$ +1 This is an interesting question. Just out of curiosity -- why don't you want to use ReLUs? $\endgroup$
    – Sycorax
    Commented Aug 3, 2016 at 16:41
  • $\begingroup$ my reasons are: 1) It seems to me that using ReLu sort of avoids the question of vanishing and exploding gradient rather than address it directly. 2) if one had an activation that one believed to be very good at some tasks rather than others, then somehow we are forced to switch to ReLu even though they might not be the right one. 3) ... (next comments) $\endgroup$ Commented Aug 3, 2016 at 16:46
  • $\begingroup$ 3) when dealing with gaussians for example, generalizing batch normalization is not trivial (should I batch normalize the euclidean norm preactivation? If I do then the input to the RBF guassian could be negative which leads to exploding gradients, but what if I instead I normalize the output of the non-linearity directly, then, we are normalizing something that already had constrained moments, since the output of RBF and in fact sigmoid (tanh) too, are restricted to a small interval [0,1] or [-1,1] ) Basically, it just seems these two ideas could be taken further, I just don't know how. $\endgroup$ Commented Aug 3, 2016 at 16:48
  • $\begingroup$ Though my reasons are based mostly on hunches, clearly, we need more research or something to deal with all those points. $\endgroup$ Commented Aug 3, 2016 at 16:49
  • 1
    $\begingroup$ Just curious if you know anything about how LSTMs or Residual networks do with saturating nonlinearities. I wonder if they only tend to work with ReLus. In the papers they both specify that the use ReLus but I wonder if that is essential to the methods or just something that gives them a boost. $\endgroup$
    – user27028
    Commented Aug 12, 2016 at 15:43

2 Answers 2


Have you looked into RMSProp? Take a look at this set of slides from Geoff Hinton:

Overview of mini-batch gradient descent

Specifically page 29, entitled 'rmsprop: A mini-batch version of rprop', although it's probably worth reading through the full set to get a fuller idea of some of the related ideas.

Also related is Yan Le Cun's No More Pesky Learning Rates

and Brandyn Webb's SMORMS3.

The main idea is to look at the sign of gradient and whether it's flip-flopping or not; if it's consistent then you want to move in that direction, and if the sign isn't flipping then whatever step you just took must be OK, provided it isn't vanishingly small, so there are ways of controlling the step size to keep it sensible and that are somewhat independent of the actual gradient.

So the short answer to how to handle vanishing or exploding gradients is simply - don't use the gradient's magnitude!

  • $\begingroup$ how do you do "so there are ways of controlling the step size to keep it sensible and that are somewhat independent of the actual gradient." Does one have to do this manually or is there an algorithm that works that does this? $\endgroup$ Commented Aug 14, 2016 at 17:24
  • $\begingroup$ There are various methods (and variants) discussed in the links, but they do all provide a method for calculating a weight update that is something other than the product of a gradient and a learning rate. $\endgroup$
    – redcalx
    Commented Aug 14, 2016 at 18:28

Some of my understandings, may not be correct.

The cause of the vanishing gradient problem is that sigmoid tanh (and RBF) saturate on both sides (-inf and inf), so it's very likely for the input of such non-linearity to fall on the saturated regions.

The effect of BN is that it "pulls" the input of the non-linearity towards a small range around 0 $N(0,1)$ as a starting point , where such non-linearities don't saturate. So I guess it will work with RBF as well.

To remove the non-linearity of ReLU, we can use the softplus funtion $\log(1+e^x)$, which is very close to ReLU, and was used in Geoffrey Hinton`s papper to explain why ReLU would work.

enter image description here

Also the residual networks or the highway networks provide another way of addressing vanishing gradients (via shortcuts). From my experience such architecture gets trained way faster than only connecting the loss to the last layer.

Moreover the difficulty of training deep networks is not solely because of the vanishing gradient, but other factors as well (e.g. the internal covariate shift). There's a recent paper layer normalization about another way of doing normalization, it doesn't say about vanishing gradients though, but maybe you'll be interested.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.