# Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$.

The reason he gives is that the outputs using $tanh$ centre around 0 rather than $sigmoid$'s 0.5, and this "makes learning for the next layer a little bit easier".

1. Why does centring the activation's output speed learning? I assume he's referring to the previous layer as learning happens during backprop?

2. Are there any other features that make $tanh$ preferable? Would the steeper gradient delay vanishing gradients?

3. Are there any situations where $sigmoid$ would be preferable?

• A sigmoid function is S-shaped (hence the name). Presumably you are talking about the logistic function $\frac{e^x}{1+e^x}$. Apart from scale and location, the two are essentially the same: $\text{logistic}(x)=\frac12 +\frac12\tanh(\frac{x}2)$. So the real choice is whether you want outputs in the interval $(-1,1)$ or the interval $(0,1)$ Feb 26, 2018 at 8:59
• double (and single) precision floats have less round-off error closer to 0. Aug 6, 2021 at 16:52

Yan LeCun and others argue in Efficient BackProp that

Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme case where all the inputs are positive. Weights to a particular node in the first weight layer are updated by an amount proportional to $$\delta x$$ where $$\delta$$ is the (scalar) error at that node and $$x$$ is the input vector (see equations (5) and (10)). When all of the components of an input vector are positive, all of the updates of weights that feed into a node will have the same sign (i.e. sign($$\delta$$)). As a result, these weights can only all decrease or all increase together for a given input pattern. Thus, if a weight vector must change direction it can only do so by zigzagging which is inefficient and thus very slow.

This is why you should normalize your inputs so that the average is zero.

The same logic applies to middle layers:

This heuristic should be applied at all layers which means that we want the average of the outputs of a node to be close to zero because these outputs are the inputs to the next layer.

Postscript @craq makes the point that this quote doesn't make sense for ReLU(x)=max(0,x) which has become a widely popular activation function. While ReLU does avoid the first zigzag problem mentioned by LeCun, it doesn't solve this second point by LeCun who says it is important to push the average to zero. I would love to know what LeCun has to say about this. In any case, there is a paper called Batch Normalization, which builds on top of the work of LeCun and offers a way to address this issue:

It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated. As each layer observes the inputs produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer.

By the way, this video by Siraj explains a lot about activation functions in 10 fun minutes.

@elkout says "The real reason that tanh is preferred compared to sigmoid (...) is that the derivatives of the tanh are larger than the derivatives of the sigmoid."

I think this is a non-issue. I never seen this being a problem in the literature. If it bothers you that one derivative is smaller than another, you can just scale it.

The logistic function has the shape $$\sigma(x)=\frac{1}{1+e^{-kx}}$$. Usually, we use $$k=1$$, but nothing forbids you from using another value for $$k$$ to make your derivatives wider, if that was your problem.

Nitpick: tanh is also a sigmoid function. Any function with a S shape is a sigmoid. What you guys are calling sigmoid is the logistic function. The reason why the logistic function is more popular is historical reasons. It has been used for a longer time by statisticians. Besides, some feel that it is more biologically plausible.

• You don't need a citation to show that $\max_x \sigma^\prime(x) < \max_x \tanh^\prime(x)$, just high-school calculus. $$\sigma^\prime(x) = \sigma(x) (1 - \sigma(x)) \le 0.25$$ We know that this is true because $0 < \sigma(x) < 1$, so you just have to maximize a concave quadratic. $$\tanh^\prime(x) = \text{sech}^2(x) = \frac{2}{\exp(x) + \exp(-x))} \le 1.0$$ which can be verified by inspection.
– Sycorax
Feb 27, 2018 at 22:32
• Apart from that I said that in most cases the derivatives of tanh are larger than the derivatives of the sigmoid. This happens mostly when we are around 0. You are welcome to have a look at this link and at the clear answers provided here question which they also state that the derivates of $\tanh$ are usually larger than the derivates of the $\text{sigmoid}$. Feb 28, 2018 at 1:21
• hang on... that sounds plausible, but if middle layers should have an average output of zero, how come ReLU works so well? Isn't that a contradiction?
– craq
Jul 25, 2019 at 22:59
• @craq, good point, I think that's a flaw in LeCun's argument indeed. I have added a link to the batch normalization paper where it discusses more about that issue and how it can be ameliorated. Unfortunately, that paper doesn't compare relu with tanh, it only compares relu with logistic (sigmoid). Jul 27, 2019 at 9:40
• How does ReLU solve the zigzagging problem? Jun 16, 2020 at 16:39

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and they also share a similar "trend". Needless to say that the $\tanh$ function is called a shifted version of the $\text{sigmoid}$ function.

The real reason that $\text{tanh}$ is preferred compared to $\text{sigmoid}$, especially when it comes to big data when you are usually struggling to find quickly the local (or global) minimum, is that the derivatives of the $\text{tanh}$ are larger than the derivatives of the $\text{sigmoid}$. In other words, you minimize your cost function faster if you use $\text{tanh}$ as an activation fuction.

But why does the hyperbolic tangent have larger derivatives? Just to give you a very simple intuition you may observe the following graph:

The fact that the range is between -1 and 1 compared to 0 and 1, makes the function to be more convenient for neural networks. Apart from that, if I use some math, I can prove that:

$$\tanh{x} = 2σ(2x)-1$$

And in general, we may prove that in most cases $\Big|\frac{\partial\tanh (x)}{\partial x}\Big| > \Big|\frac{\partial\text{σ} (x)}{\partial x}\Big|$.

• So why would Prof. Ng say that it's an advantage to have the output of the function averaging around $0$? Feb 26, 2018 at 14:48
• It's not the fact that the average is around 0 that makes $\tanh$ faster. It's the fact that being around zero means that the range is also grater (compared to being around 0.5 in the case of $\text{sigmoid}$), which leads to larger derivatives, which almost always leads to faster convergence to the minimum. I hope that it is clear now. Ng is right that we prefer the $\tanh$ function because it is centered around 0, but he just didn't provide the complete justification. Feb 26, 2018 at 14:53
• Zero-centering is more important than $2x$ ratio, because it skews the distribution of activations and that hurts the performance. If you take sigmoid(x) - 0.5 and $2x$ smaller learning rate, it will learn on par with tanh. Feb 26, 2018 at 21:22
• @Maxim Which "it" skews the distribution of activations, zero-centering or $2x$? If zero-centering is a Good Thing, I still don't feel that the "why" of that has been answered. Feb 27, 2018 at 4:18
• From the 1st order derivative curves of sigmoid and tanh, we can see that only within range (-2, 2), tanh has larger derivative than sigmoid. So by "most cases", we should prove that most of the time, the input values of activation function should fall within this range, right? And how do we prove that? Aug 25, 2020 at 15:00

It all essentially depends on the derivatives of the activation function, the main problem with the sigmoid function is that the max value of its derivative is 0.25, this means that the update of the values of W and b will be small.

The tanh function on the other hand, has a derivativ of up to 1.0, making the updates of W and b much larger.

This makes the tanh function almost always better as an activation function (for hidden layers) rather than the sigmoid function.

To prove this myself (at least in a simple case), I coded a simple neural network and used sigmoid, tanh and relu as activation functions, then I plotted how the error value evolved and this is what I got.

The full notebook I wrote is here https://www.kaggle.com/moriano/a-showcase-of-how-relus-can-speed-up-the-learning

If it helps, here are the charts of the derivatives of the tanh function and the sigmoid one (pay attention to the vertical axis!)

• (-1) Although this is an interesting idea, it doesn't stand on it's own. In particular, most optimization methods used for DL/NN are first order gradient methods, which have a learning rate $\alpha$. If the max derivative with regards to one activation function is too small, one could easily just increase the learning rate. Jul 27, 2019 at 15:33
• Don't you run the risk of not having a stable learning curve with a higher learning rate? Jul 27, 2019 at 21:18
• Well, if the derivatives are more stable, then increasing the learning rate is less likely to destablize the estimation. Jul 27, 2019 at 22:54
• That's a fair point, do you have a link where I could learn more of this? Jul 28, 2019 at 23:10
• it might be useful if you tried a custom function with Tanh/logistic so they have same zero mean. Aug 9, 2023 at 9:04

Andrew Ng says that using the logistic function (commonly know as sigmoid) really only makes sense in the final layer of a binary classification network.

As the output of the network is expected to be between $0$ and $1$, the logistic is a perfect choice as it's range is exactly $(0, 1)$. No scaling and shifting of $tanh$ required.

• For the output, the logistic function makes sense if you want to produce probabilities, we can all agree on that. What is being discussed is why tanh is preferred over the logistic function as an activation for the middle layers. Jul 27, 2019 at 9:44
• How do you know that's what the OP intended? It seems he was asking a general question. Jul 27, 2019 at 16:39

And especially when you are solving an Ordinary Differential Equation with a stiff gradient.

tanh(x) converges earlier than sigmoids of all types.

– Community Bot
Aug 9, 2023 at 8:55

Zero mean assumption relates to the curvature of the error surface.

The whitening argument (https://stats.stackexchange.com/a/330885/27556) is a standard issue with gradient descent. just consider linear regression with gradient descent - the curvature of the error surface is given by the covariance of the inputs (including bias term). [ for a nonlinear model such as a neural net, I would argue you have the same issues]

Zero mean ensures you have no covariance with the bias term ( and the other non zero mean inputs).

If you have covariance with a lot of inputs, then you create a narrow valley shape (in our case across the bias/constant input direction). This is hard for gradient descent to handle (because you would want a bigger learning rate along the valley and a shorter one across the valley), gradient descent has a single learning rate, so you are forced to have a very low learning rate (making slow progress along the valley) to avoid jumping from peak to peak of the valley. Having different learning rates in each direction is essentially what second order methods do, but they are too memory intensive for large parameter models such as neural networks.

Generally, the non-zero centered activation function restricts the movement of parameters over the surface area in some specific directions. Which makes the training slower because it needs mush steps to move from the initial point to the minimum point with these restricted movements. For more details watch only 7 min from this video starting from time 8:43.

• This is very similar to the answer here. Please do not post duplicate answers. If you believe a question is completely answered by an answer elsewhere, flag / vote to close that question as a duplicate of the other. If it isn't completely answered by the other answer, customize your answer here to be more specific to the question. Aug 21, 2020 at 1:00