# tanh activation function vs sigmoid activation function

The tanh activation function is:

$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:

• Does it really matter between using those two activation functions (tanh vs. sigma)?
• Which function is better in which cases?
• $\textrm{tanh}(x) = 2\sigma(2x) - 1$ Sep 23, 2014 at 15:13
• Deep Neural Networks have moved on. The current preference is the RELU function. Oct 4, 2017 at 4:35
• @PaulNord Both tanh and sigmoids are still used in conjunction with other activations like RELU, depends what you're trying to do. Oct 27, 2017 at 17:36
• @RomanShapovalov so we can say that having 2 layers sigmoid + linear are at least as powerful as a layer with tanh? Jul 31, 2022 at 17:35

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):

1. Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]

1. Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
• I have small doubt in the paper you suggested. In page 14, "When MLP have shared weights (eg:Convolutional nets), Learning rate should be choosen in such a way that, it is proportional to square root of no. of connections sharing the weight." Can you please explain why? Jun 9, 2014 at 13:06
• this question has already been answered here stats.stackexchange.com/questions/47590/… Jun 9, 2014 at 13:55
• That is a very general question. Long story short: the cost function determines what the neural network should do: classification or regression and how. If you could get a copy of "Neural Networks for Pattern Recognition" by Christopher Bishop that'd be great. Also "Machine Learning" by Mitchell gives you a good explanation at a more basic level. Jun 10, 2014 at 9:30
• I am sorry, Satya, I am usually quite busy during the week. How do you normalize your data exactly? en.wikipedia.org/wiki/Whitening_transformation I am not really sure what your problem can be. The easiest way is to substract the mean and then equalize with the covariance matrix. Evtl. you need to add some component for high frequencies (see ZCA transform in the reference above) Jun 11, 2014 at 21:27
• Thanks a lot juampa. You are really helping me a lot. Suggested reading are very good. I am actually doing a project on climate data mining. 50% of my input features are temperature(range 200K-310K) and 50% of my input features are pressure values(range 50000pa to 100000pa). I am doing whitening. Before pca, is there any need to normalize it... If yes, how should I normalize it? Should I normalize before subtracting by mean or after subtracting by mean? I am getting different results if I am normalizing by different methods... Jun 12, 2014 at 10:22

Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got. This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1]. This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)). For input between [0,1], we have derivative between [0.20, 0.25]. Apparently the tanh function provides stronger gradients.

• Another way of looking at this is that σ(2x) is the same as σ(x), but with a horizontal stretch applied, scale factor 1/2 (i.e. it's the same graph but with everything squashed in towards the y axis). When you squash it in, the slope gets steeper Mar 3, 2017 at 16:05
• I don't see why this would make any difference. The scale and squashing will be random for each node, and (with offsets and weights on input and output) both will be universal approximators, converging to the same result. Nov 25, 2018 at 16:31
• @endolith The question is not one of efficacy, but of efficiency. A model that converges to the same result but takes a hundred times as long to do so is not a model I want to be using Feb 10, 2020 at 6:28
• @TheEnvironmentalist I mean that with bias, they're exactly the same function, so efficacy and efficiency are the same. Feb 10, 2020 at 17:30
• Depends on how you're using these activation functions, but you can just increase the learning rate, or multiply loss/layer outputs by some factor, and then convergence will be similar. So in that respect you are correct @endolith Aug 8, 2020 at 1:54

Generally speaking, $$\tanh$$ has two main advantages over a sigmoid function:

1. It has a slightly bigger derivative than the sigmoid (at least for the area around 0), which helps it to cope a bit better with the “vanishing gradients” problem of deep neural networks. Here is a plot of the derivatives of both functions: 1. It is symmetric around 0, which helps it to avoid the “bias shift” problem that sigmoid suffer from (which causes the weight vectors to move in diagonals, or “zig-zag”, which slows down learning).

Sigmoid has 1 main advantage over $$\tanh$$, which is that it can represent a binary probability - hence can be used as the output of the final layer in binary classification problems.

You can check out this video I made on YouTube which explains a bit further about these problems.

Elaboration on the bias shift problem:

Consider a case of activation functions like Sigmoids which only output positive values. Now let’s focus on a single layer $$a_l$$. Let’s look at the weight vector associated with the first next neuron: $$z_{(l+1),1}=W_{l,1}\cdot a_l + b_{l,1}$$. The gradient w.r.t. this vector will be (according to the chain rule) $$a_l \cdot \frac{\partial \mathcal L}{\partial z_{(l+1),1}}$$. That is the gradient up to $$z_{(l+1),1}$$ (which is a scalar) times the gradient of $$z_{(l+1),1}$$ w.r.t. $$W_{l1}$$ which is just $$a_l$$.

We know that the $$a_l$$ neurons are all $$\ge 0$$, so this $$W_{l1}$$ vector updates depend on $$sign(\frac{\partial \mathcal L}{\partial z_{(l+1),1}}).$$

This means that the vector either increase or decrease for all elements $$\Rightarrow$$ it can only move in Zig-Zag / diagonals, which is not very efficient. This is sometimes called the “bias shift” problem. It also happens when the activations output values which are far from 0 (though to a less extent).