Why is tanh almost always better than sigmoid as an activation function?

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$.

The reason he gives is that the outputs using $tanh$ centre around 0 rather than $sigmoid$'s 0.5, and this "makes learning for the next layer a little bit easier".

1. Why does centring the activation's output speed learning? I assume he's referring to the previous layer as learning happens during backprop?

2. Are there any other features that make $tanh$ preferable? Would the steeper gradient delay vanishing gradients?

3. Are there any situations where $sigmoid$ would be preferable?

• A sigmoid function is S-shaped (hence the name). Presumably you are talking about the logistic function $\frac{e^x}{1+e^x}$. Apart from scale and location, the two are essentially the same: $\text{logistic}(x)=\frac12 +\frac12\tanh(\frac{x}2)$. So the real choice is whether you want outputs in the interval $(-1,1)$ or the interval $(0,1)$ – Henry Feb 26 '18 at 8:59

Yan LeCun and others argue in Efficient BackProp that

Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme case where all the inputs are positive. Weights to a particular node in the first weight layer are updated by an amount proportional to $$\delta x$$ where $$\delta$$ is the (scalar) error at that node and $$x$$ is the input vector (see equations (5) and (10)). When all of the components of an input vector are positive, all of the updates of weights that feed into a node will have the same sign (i.e. sign($$\delta$$)). As a result, these weights can only all decrease or all increase together for a given input pattern. Thus, if a weight vector must change direction it can only do so by zigzagging which is inefficient and thus very slow.

This is why you should normalize your inputs so that the average is zero.

The same logic applies to middle layers:

This heuristic should be applied at all layers which means that we want the average of the outputs of a node to be close to zero because these outputs are the inputs to the next layer.

Postscript @craq makes the point that this quote doesn't make sense for ReLU(x)=max(0,x) which has become a widely popular activation function. While ReLU does avoid the first zigzag problem mentioned by LeCun, it doesn't solve this second point by LeCun who says it is important to push the average to zero. I would love to know what LeCun has to say about this. In any case, there is a paper called Batch Normalization, which builds on top of the work of LeCun and offers a way to address this issue:

It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated. As each layer observes the inputs produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer.

By the way, this video by Siraj explains a lot about activation functions in 10 fun minutes.

@elkout says "The real reason that tanh is preferred compared to sigmoid (...) is that the derivatives of the tanh are larger than the derivatives of the sigmoid."

I think this is a non-issue. I never seen this being a problem in the literature. If it bothers you that one derivative is smaller than another, you can just scale it.

The logistic function has the shape $$\sigma(x)=\frac{1}{1+e^{-kx}}$$. Usually, we use $$k=1$$, but nothing forbids you from using another value for $$k$$ to make your derivatives wider, if that was your problem.

Nitpick: tanh is also a sigmoid function. Any function with a S shape is a sigmoid. What you guys are calling sigmoid is the logistic function. The reason why the logistic function is more popular is historical reasons. It has been used for a longer time by statisticians. Besides, some feel that it is more biologically plausible.

• You don't need a citation to show that $\max_x \sigma^\prime(x) < \max_x \tanh^\prime(x)$, just high-school calculus. $$\sigma^\prime(x) = \sigma(x) (1 - \sigma(x)) \le 0.25$$ We know that this is true because $0 < \sigma(x) < 1$, so you just have to maximize a concave quadratic. $$\tanh^\prime(x) = \text{sech}^2(x) = \frac{2}{\exp(x) + \exp(-x))} \le 1.0$$ which can be verified by inspection. – Sycorax says Reinstate Monica Feb 27 '18 at 22:32
• Apart from that I said that in most cases the derivatives of tanh are larger than the derivatives of the sigmoid. This happens mostly when we are around 0. You are welcome to have a look at this link and at the clear answers provided here question which they also state that the derivates of $\tanh$ are usually larger than the derivates of the $\text{sigmoid}$. – ekoulier Feb 28 '18 at 1:21
• hang on... that sounds plausible, but if middle layers should have an average output of zero, how come ReLU works so well? Isn't that a contradiction? – craq Jul 25 '19 at 22:59
• @ekoulier, the derivative of $\text{tanh}$ being larger than $\text{sigmoid}$ is a non-issue. You can just scale it if it bothers you. – Ricardo Cruz Jul 27 '19 at 9:37
• @craq, good point, I think that's a flaw in LeCun's argument indeed. I have added a link to the batch normalization paper where it discusses more about that issue and how it can be ameliorated. Unfortunately, that paper doesn't compare relu with tanh, it only compares relu with logistic (sigmoid). – Ricardo Cruz Jul 27 '19 at 9:40

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and they also share a similar "trend". Needless to say that the $\tanh$ function is called a shifted version of the $\text{sigmoid}$ function.

The real reason that $\text{tanh}$ is preferred compared to $\text{sigmoid}$, especially when it comes to big data when you are usually struggling to find quickly the local (or global) minimum, is that the derivatives of the $\text{tanh}$ are larger than the derivatives of the $\text{sigmoid}$. In other words, you minimize your cost function faster if you use $\text{tanh}$ as an activation fuction.

But why does the hyperbolic tangent have larger derivatives? Just to give you a very simple intuition you may observe the following graph:

The fact that the range is between -1 and 1 compared to 0 and 1, makes the function to be more convenient for neural networks. Apart from that, if I use some math, I can prove that:

$$\tanh{x} = 2σ(2x)-1$$

And in general, we may prove that in most cases $\Big|\frac{\partial\tanh (x)}{\partial x}\Big| > \Big|\frac{\partial\text{σ} (x)}{\partial x}\Big|$.

• So why would Prof. Ng say that it's an advantage to have the output of the function averaging around $0$? – Tom Hale Feb 26 '18 at 14:48
• It's not the fact that the average is around 0 that makes $\tanh$ faster. It's the fact that being around zero means that the range is also grater (compared to being around 0.5 in the case of $\text{sigmoid}$), which leads to larger derivatives, which almost always leads to faster convergence to the minimum. I hope that it is clear now. Ng is right that we prefer the $\tanh$ function because it is centered around 0, but he just didn't provide the complete justification. – ekoulier Feb 26 '18 at 14:53
• Zero-centering is more important than $2x$ ratio, because it skews the distribution of activations and that hurts the performance. If you take sigmoid(x) - 0.5 and $2x$ smaller learning rate, it will learn on par with tanh. – Maxim Feb 26 '18 at 21:22
• @Maxim Which "it" skews the distribution of activations, zero-centering or $2x$? If zero-centering is a Good Thing, I still don't feel that the "why" of that has been answered. – Tom Hale Feb 27 '18 at 4:18

Andrew Ng says that using the logistic function (commonly know as sigmoid) really only makes sense in the final layer of a binary classification network.

As the output of the network is expected to be between $0$ and $1$, the logistic is a perfect choice as it's range is exactly $(0, 1)$. No scaling and shifting of $tanh$ required.

• For the output, the logistic function makes sense if you want to produce probabilities, we can all agree on that. What is being discussed is why tanh is preferred over the logistic function as an activation for the middle layers. – Ricardo Cruz Jul 27 '19 at 9:44
• How do you know that's what the OP intended? It seems he was asking a general question. – Tom Hale Jul 27 '19 at 16:39

It all essentially depends on the derivatives of the activation function, the main problem with the sigmoid function is that the max value of its derivative is 0.25, this means that the update of the values of W and b will be small.

The tanh function on the other hand, has a derivativ of up to 1.0, making the updates of W and b much larger.

This makes the tanh function almost always better as an activation function (for hidden layers) rather than the sigmoid function.

To prove this myself (at least in a simple case), I coded a simple neural network and used sigmoid, tanh and relu as activation functions, then I plotted how the error value evolved and this is what I got.

The full notebook I wrote is here https://www.kaggle.com/moriano/a-showcase-of-how-relus-can-speed-up-the-learning

If it helps, here are the charts of the derivatives of the tanh function and the sigmoid one (pay attention to the vertical axis!)

• (-1) Although this is an interesting idea, it doesn't stand on it's own. In particular, most optimization methods used for DL/NN are first order gradient methods, which have a learning rate $\alpha$. If the max derivative with regards to one activation function is too small, one could easily just increase the learning rate. – Cliff AB Jul 27 '19 at 15:33
• Don't you run the risk of not having a stable learning curve with a higher learning rate? – Juan Antonio Gomez Moriano Jul 27 '19 at 21:18
• Well, if the derivatives are more stable, then increasing the learning rate is less likely to destablize the estimation. – Cliff AB Jul 27 '19 at 22:54
• That's a fair point, do you have a link where I could learn more of this? – Juan Antonio Gomez Moriano Jul 28 '19 at 23:10