# What does having a stronger gradient mean intuitively when talking about various activation functions?

I was reading about the various activation functions that are available to choose from. For example:

• Sigmoid activation function
• Tanh activation function
• Relu activation function
• etc..

I came across a post here, explaining why Tanh functions are better than sigmoid functions. One of the point that has been mentioned says:

What does this statement mean? What are strong gradients? How do stronger gradients help in the learning process? It will be helpful if it could be explained with the help of a real-world example.

I also came across the following graph that describes the gradients for some activation functions.

but could interpret nothing from it.

• ..Larger gradients mean that the optimization converges faster (assuming constant learning rate)... Could you explain this? – Suhail Gupta Apr 6 at 11:09
• Let $L_1$ be loss function of network using tanh as activation (implies larger gradient), $L_2$ loss of network with sigmoid, then $w := w - \alpha \partial L_1 / \partial w$ performs “bigger update” than the same process but with $L_2$, assuming the networks are in the same state (all parameters are the same at that time). – pixelneo Apr 6 at 13:00