# tanh activation function vs sigmoid activation function

The tanh activation function is:

$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:

• Does it really matter between using those two activation functions (tanh vs. sigma)?
• Which function is better in which cases?
• $\textrm{tanh}(x) = 2\sigma(2x) - 1$ Sep 23 '14 at 15:13
• Deep Neural Networks have moved on. The current preference is the RELU function. Oct 4 '17 at 4:35
• @PaulNord Both tanh and sigmoids are still used in conjunction with other activations like RELU, depends what you're trying to do. Oct 27 '17 at 17:36

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):

1. Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]

1. Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
• I have small doubt in the paper you suggested. In page 14, "When MLP have shared weights (eg:Convolutional nets), Learning rate should be choosen in such a way that, it is proportional to square root of no. of connections sharing the weight." Can you please explain why? Jun 9 '14 at 13:06
• this question has already been answered here stats.stackexchange.com/questions/47590/… Jun 9 '14 at 13:55
• That is a very general question. Long story short: the cost function determines what the neural network should do: classification or regression and how. If you could get a copy of "Neural Networks for Pattern Recognition" by Christopher Bishop that'd be great. Also "Machine Learning" by Mitchell gives you a good explanation at a more basic level. Jun 10 '14 at 9:30
• I am sorry, Satya, I am usually quite busy during the week. How do you normalize your data exactly? en.wikipedia.org/wiki/Whitening_transformation I am not really sure what your problem can be. The easiest way is to substract the mean and then equalize with the covariance matrix. Evtl. you need to add some component for high frequencies (see ZCA transform in the reference above) Jun 11 '14 at 21:27
• Thanks a lot juampa. You are really helping me a lot. Suggested reading are very good. I am actually doing a project on climate data mining. 50% of my input features are temperature(range 200K-310K) and 50% of my input features are pressure values(range 50000pa to 100000pa). I am doing whitening. Before pca, is there any need to normalize it... If yes, how should I normalize it? Should I normalize before subtracting by mean or after subtracting by mean? I am getting different results if I am normalizing by different methods... Jun 12 '14 at 10:22

Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got. This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1]. This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)). For input between [0,1], we have derivative between [0.20, 0.25]. Apparently the tanh function provides stronger gradients.

• Another way of looking at this is that σ(2x) is the same as σ(x), but with a horizontal stretch applied, scale factor 1/2 (i.e. it's the same graph but with everything squashed in towards the y axis). When you squash it in, the slope gets steeper Mar 3 '17 at 16:05
• I don't see why this would make any difference. The scale and squashing will be random for each node, and (with offsets and weights on input and output) both will be universal approximators, converging to the same result. Nov 25 '18 at 16:31
• @endolith The question is not one of efficacy, but of efficiency. A model that converges to the same result but takes a hundred times as long to do so is not a model I want to be using Feb 10 '20 at 6:28
• @TheEnvironmentalist I mean that with bias, they're exactly the same function, so efficacy and efficiency are the same. Feb 10 '20 at 17:30
• Depends on how you're using these activation functions, but you can just increase the learning rate, or multiply loss/layer outputs by some factor, and then convergence will be similar. So in that respect you are correct @endolith Aug 8 '20 at 1:54