0
$\begingroup$

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $\tanh(n x)$ then the maximum possible gradient is $n$. Thus if $n > 1$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?

$\endgroup$
0
$\begingroup$

The MAXIMUM possible gradient would be $n$ if the $tanh(nx) = 0$ (for $x = 0$).

But it doesn't tell you anything about what would be the gradient on average, and this is what it's all about - if you move from that zero just so slightly, then the gradient vanishes much faster than it had with a plain $tanh(x)$, and you end up with much more severe gradient vanishing than before.

$\endgroup$
  • $\begingroup$ perhaps - but with tanh(x) the product of gradients necessarilly tends to 0, whearas here, though it may on average go to zero, it can take any value (up to $2^n$ where $n$ is the number of hidden layers) - are you aware of any research exploring this? $\endgroup$ – Zephyr May 8 at 16:56
0
$\begingroup$

I've been thinking about this for a while and just found the answer to my own question:

When we replace 𝑡𝑎𝑛ℎ(𝑥) with 𝑡𝑎𝑛ℎ(𝑛𝑥) as an activation function we have changed nothing about how the activation function works.

All we have done is rescaled all the weights of the network - which we are free to do arbitrarily.

The only stage where $n$ should come into the calculation is the normalization of gradients in the initialization - but a well-converged network should lose any information of the initialization stage (in the ideal case at least).

$\endgroup$
-1
$\begingroup$

Very interesting idea, but I'm worried about the other type of problem, overflow. As long as we keep multiplying numbers that are less than 1, we won't ever get a back-prop that's larger than 1. But with >1, and especially deep networks, I can imagine you could quickly overflow.

$\endgroup$
  • $\begingroup$ but the derivative of rectified units can be larger than one $\endgroup$ – bibliolytic May 7 at 15:08
  • $\begingroup$ max(0,x) has a derivative of 0 or 1. $\endgroup$ – Anonymous Emu May 7 at 15:10
  • $\begingroup$ ...yep got confused sorry $\endgroup$ – bibliolytic May 7 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.