tanh activation function vs sigmoid activation function

The tanh activation function is:

$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:

• Does it really matter between using those two activation functions (tanh vs. sigma)?
• Which function is better in which cases?
• $\textrm{tanh}(x) = 2\sigma(2x) - 1$ – Roman Shapovalov Sep 23 '14 at 15:13
• Deep Neural Networks have moved on. The current preference is the RELU function. – Paul Nord Oct 4 '17 at 4:35
• @PaulNord Both tanh and sigmoids are still used in conjunction with other activations like RELU, depends what you're trying to do. – Tahlor Oct 27 '17 at 17:36

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):

1. Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]

1. Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
• I have small doubt in the paper you suggested. In page 14, "When MLP have shared weights (eg:Convolutional nets), Learning rate should be choosen in such a way that, it is proportional to square root of no. of connections sharing the weight." Can you please explain why? – satya Jun 9 '14 at 13:06
• this question has already been answered here stats.stackexchange.com/questions/47590/… – jpmuc Jun 9 '14 at 13:55
• That is a very general question. Long story short: the cost function determines what the neural network should do: classification or regression and how. If you could get a copy of "Neural Networks for Pattern Recognition" by Christopher Bishop that'd be great. Also "Machine Learning" by Mitchell gives you a good explanation at a more basic level. – jpmuc Jun 10 '14 at 9:30
• I am sorry, Satya, I am usually quite busy during the week. How do you normalize your data exactly? en.wikipedia.org/wiki/Whitening_transformation I am not really sure what your problem can be. The easiest way is to substract the mean and then equalize with the covariance matrix. Evtl. you need to add some component for high frequencies (see ZCA transform in the reference above) – jpmuc Jun 11 '14 at 21:27
• Thanks a lot juampa. You are really helping me a lot. Suggested reading are very good. I am actually doing a project on climate data mining. 50% of my input features are temperature(range 200K-310K) and 50% of my input features are pressure values(range 50000pa to 100000pa). I am doing whitening. Before pca, is there any need to normalize it... If yes, how should I normalize it? Should I normalize before subtracting by mean or after subtracting by mean? I am getting different results if I am normalizing by different methods... – satya Jun 12 '14 at 10:22

Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got. This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1]. This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)). For input between [0,1], we have derivative between [0.20, 0.25]. Apparently the tanh function provides stronger gradients.

• Another way of looking at this is that σ(2x) is the same as σ(x), but with a horizontal stretch applied, scale factor 1/2 (i.e. it's the same graph but with everything squashed in towards the y axis). When you squash it in, the slope gets steeper – rbennett485 Mar 3 '17 at 16:05
• I don't see why this would make any difference. The scale and squashing will be random for each node, and (with offsets and weights on input and output) both will be universal approximators, converging to the same result. – endolith Nov 25 '18 at 16:31

protected by kjetil b halvorsenJul 11 '18 at 12:38

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?