tanh activation function vs sigmoid activation function The tanh activation function is: 
$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$
Where $\sigma(x)$, the sigmoid function, is defined as:
 $$\sigma(x) = \frac{e^x}{1 + e^x}$$.
Questions: 


*

*Does it really matter between using those two activation functions
(tanh vs. sigma)?

*Which function is better in which cases?

 A: Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got.
This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1].

This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)).
For input between [0,1], we have derivative between [0.20, 0.25].

Apparently the tanh function provides stronger gradients.
A: Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.
There are two reasons for that choice (assuming you have normalized your data, and this is very important):


*

*Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].


The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]


*Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.

