0
$\begingroup$

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $\tanh(n x)$ then the maximum possible gradient is $n$. Thus if $n > 1$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?

$\endgroup$
0
$\begingroup$

The MAXIMUM possible gradient would be $n$ if the $tanh(nx) = 0$ (for $x = 0$).

But it doesn't tell you anything about what would be the gradient on average, and this is what it's all about - if you move from that zero just so slightly, then the gradient vanishes much faster than it had with a plain $tanh(x)$, and you end up with much more severe gradient vanishing than before.

$\endgroup$
1
  • $\begingroup$ perhaps - but with tanh(x) the product of gradients necessarilly tends to 0, whearas here, though it may on average go to zero, it can take any value (up to $2^n$ where $n$ is the number of hidden layers) - are you aware of any research exploring this? $\endgroup$
    – zephyr
    May 8 '19 at 16:56
0
$\begingroup$

I've been thinking about this for a while and just found the answer to my own question:

When we replace ๐‘ก๐‘Ž๐‘›โ„Ž(๐‘ฅ) with ๐‘ก๐‘Ž๐‘›โ„Ž(๐‘›๐‘ฅ) as an activation function we have changed nothing about how the activation function works.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily.

The only stage where $n$ should come into the calculation is the normalization of gradients in the initialization - but a well-converged network should lose any information of the initialization stage (in the ideal case at least).

$\endgroup$
-1
$\begingroup$

Very interesting idea, but I'm worried about the other type of problem, overflow. As long as we keep multiplying numbers that are less than 1, we won't ever get a back-prop that's larger than 1. But with >1, and especially deep networks, I can imagine you could quickly overflow.

$\endgroup$
3
  • $\begingroup$ but the derivative of rectified units can be larger than one $\endgroup$ May 7 '19 at 15:08
  • $\begingroup$ max(0,x) has a derivative of 0 or 1. $\endgroup$ May 7 '19 at 15:10
  • $\begingroup$ ...yep got confused sorry $\endgroup$ May 7 '19 at 15:11

Your Answer

By clicking โ€œPost Your Answerโ€, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.