# What does having a stronger gradient mean intuitively when talking about various activation functions?

I was reading about the various activation functions that are available to choose from. For example:

• Sigmoid activation function
• Tanh activation function
• Relu activation function
• etc..

I came across a post here, explaining why Tanh functions are better than sigmoid functions. One of the point that has been mentioned says:

Tanh function provided stronger gradients

What does this statement mean? What are strong gradients? How do stronger gradients help in the learning process? It will be helpful if it could be explained with the help of a real-world example.

I also came across the following graph that describes the gradients for some activation functions. but could interpret nothing from it.

## 1 Answer

It means that the derivatives are bigger. Larger gradients mean that the optimization converges faster (assuming constant learning rate). If you look at graph of sigmoid and tanh, you can see that tanh is much “steeper” around 0, providing larger gradients.

• ..Larger gradients mean that the optimization converges faster (assuming constant learning rate)... Could you explain this? – Suhail Gupta Apr 6 at 11:09
• Let $L_1$ be loss function of network using tanh as activation (implies larger gradient), $L_2$ loss of network with sigmoid, then $w := w - \alpha \partial L_1 / \partial w$ performs “bigger update” than the same process but with $L_2$, assuming the networks are in the same state (all parameters are the same at that time). – pixelneo Apr 6 at 13:00
• Okay. Maybe you would want to include this explanation in your answer :) – Suhail Gupta Apr 6 at 13:22