0
$\begingroup$

I was reading about the various activation functions that are available to choose from. For example:

  • Sigmoid activation function
  • Tanh activation function
  • Relu activation function
  • etc..

I came across a post here, explaining why Tanh functions are better than sigmoid functions. One of the point that has been mentioned says:

Tanh function provided stronger gradients

What does this statement mean? What are strong gradients? How do stronger gradients help in the learning process? It will be helpful if it could be explained with the help of a real-world example.

I also came across the following graph that describes the gradients for some activation functions.

enter image description here

but could interpret nothing from it.

$\endgroup$
2
$\begingroup$

It means that the derivatives are bigger. Larger gradients mean that the optimization converges faster (assuming constant learning rate). If you look at graph of sigmoid and tanh, you can see that tanh is much “steeper” around 0, providing larger gradients.

$\endgroup$
  • $\begingroup$ ..Larger gradients mean that the optimization converges faster (assuming constant learning rate)... Could you explain this? $\endgroup$ – Suhail Gupta Apr 6 at 11:09
  • 1
    $\begingroup$ Let $L_1$ be loss function of network using tanh as activation (implies larger gradient), $L_2$ loss of network with sigmoid, then $w := w - \alpha \partial L_1 / \partial w$ performs “bigger update” than the same process but with $L_2$, assuming the networks are in the same state (all parameters are the same at that time). $\endgroup$ – pixelneo Apr 6 at 13:00
  • $\begingroup$ Okay. Maybe you would want to include this explanation in your answer :) $\endgroup$ – Suhail Gupta Apr 6 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.