0
$\begingroup$

I was reading about the various activation functions that are available to choose from. For example:

  • Sigmoid activation function
  • Tanh activation function
  • Relu activation function
  • etc..

I came across a post here, explaining why Tanh functions are better than sigmoid functions. One of the point that has been mentioned says:

Tanh function provided stronger gradients

What does this statement mean? What are strong gradients? How do stronger gradients help in the learning process? It will be helpful if it could be explained with the help of a real-world example.

I also came across the following graph that describes the gradients for some activation functions.

enter image description here

but could interpret nothing from it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

It means that the derivatives are bigger. Larger gradients mean that the optimization converges faster (assuming constant learning rate). If you look at graph of sigmoid and tanh, you can see that tanh is much “steeper” around 0, providing larger gradients.

Improve this answer
$\endgroup$
3
  • $\begingroup$ ..Larger gradients mean that the optimization converges faster (assuming constant learning rate)... Could you explain this? $\endgroup$
    – Suhail Gupta
    Apr 6, 2019 at 11:09
  • 2
    $\begingroup$ Let $L_1$ be loss function of network using tanh as activation (implies larger gradient), $L_2$ loss of network with sigmoid, then $w := w - \alpha \partial L_1 / \partial w$ performs “bigger update” than the same process but with $L_2$, assuming the networks are in the same state (all parameters are the same at that time). $\endgroup$
    – pixelneo
    Apr 6, 2019 at 13:00
  • $\begingroup$ Okay. Maybe you would want to include this explanation in your answer :) $\endgroup$
    – Suhail Gupta
    Apr 6, 2019 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.