# Can the vanishing gradient problem be solved by multiplying the input of tanh with a coefficient?

To my understanding, the vanishing gradient problem occurs when training neural networks when the gradient of each activation function is less than 1 such that when corrections are back-propagated through many layers, the product of these gradients becomes very small

I know there are other solutions like a rectifier activation function, but my question is why we could not simply use a variation of the often used tanh function.

If the activation function was of the form $$\tanh(n x)$$ then the maximum possible gradient is $$n$$. Thus if $$n > 1$$ we no longer have a case where the product of gradients necessarily goes to 0.

Is there some reason why such an activation function would otherwise fail?

The MAXIMUM possible gradient would be $$n$$ if the $$tanh(nx) = 0$$ (for $$x = 0$$).

But it doesn't tell you anything about what would be the gradient on average, and this is what it's all about - if you move from that zero just so slightly, then the gradient vanishes much faster than it had with a plain $$tanh(x)$$, and you end up with much more severe gradient vanishing than before.

• perhaps - but with tanh(x) the product of gradients necessarilly tends to 0, whearas here, though it may on average go to zero, it can take any value (up to $2^n$ where $n$ is the number of hidden layers) - are you aware of any research exploring this? – Zephyr May 8 at 16:56

The only stage where $$n$$ should come into the calculation is the normalization of gradients in the initialization - but a well-converged network should lose any information of the initialization stage (in the ideal case at least).