Why is tanh almost always better than sigmoid as an activation function?

Question

In Andrew Ng's Neural Networks and Deep Learning course on Coursera he says that using $tanh$ is almost always preferable to using $sigmoid$.

The reason he gives is that the outputs using $tanh$ centre around 0 rather than $sigmoid$'s 0.5, and this "makes learning for the next layer a little bit easier".

1. Why does centring the activation's output speed learning? I assume he's referring to the previous layer as learning happens during backprop?

2. Are there any other features that make $tanh$ preferable? Would the steeper gradient delay vanishing gradients?

3. Are there any situations where $sigmoid$ would be preferable?

Math-light, intuitive answers preferred.

Answer 1

Yan LeCun and others argue in Efficient BackProp that

Convergence is usually faster if the average of each input variable over the training set is close to zero. To see this, consider the extreme case where all the inputs are positive. Weights to a particular node in the first weight layer are updated by an amount proportional to $\delta x$ where $\delta$ is the (scalar) error at that node and $x$ is the input vector (see equations (5) and (10)). When all of the components of an input vector are positive, all of the updates of weights that feed into a node will have the same sign (i.e. sign($\delta$)). As a result, these weights can only all decrease or all increase together for a given input pattern. Thus, if a weight vector must change direction it can only do so by zigzagging which is inefficient and thus very slow.

This is why you should normalize your inputs so that the average is zero.

The same logic applies to middle layers:

This heuristic should be applied at all layers which means that we want the average of the outputs of a node to be close to zero because these outputs are the inputs to the next layer.

Postscript @craq makes the point that this quote doesn't make sense for ReLU(x)=max(0,x) which has become a widely popular activation function. While ReLU does avoid the first zigzag problem mentioned by LeCun, it doesn't solve this second point by LeCun who says it is important to push the average to zero. I would love to know what LeCun has to say about this. In any case, there is a paper called Batch Normalization, which builds on top of the work of LeCun and offers a way to address this issue:

It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated. As each layer observes the inputs produced by the layers below, it would be advantageous to achieve the same whitening of the inputs of each layer.

---

By the way, this video by Siraj explains a lot about activation functions in 10 fun minutes.

---

@elkout says "The real reason that tanh is preferred compared to sigmoid (...) is that the derivatives of the tanh are larger than the derivatives of the sigmoid."

I think this is a non-issue. I never seen this being a problem in the literature. If it bothers you that one derivative is smaller than another, you can just scale it.

The logistic function has the shape $\sigma(x)=\frac{1}{1+e^{-kx}}$. Usually, we use $k=1$, but nothing forbids you from using another value for $k$ to make your derivatives wider, if that was your problem.

---

Nitpick: tanh is also a sigmoid function. Any function with a S shape is a sigmoid. What you guys are calling sigmoid is the logistic function. The reason why the logistic function is more popular is historical reasons. It has been used for a longer time by statisticians. Besides, some feel that it is more biologically plausible.

Answer 2

It's not that it is necessarily better than $\text{sigmoid}$. In other words, it's not the center of an activation fuction that makes it better. And the idea behind both functions is the same, and they also share a similar "trend". Needless to say that the $\tanh$ function is called a shifted version of the $\text{sigmoid}$ function.

The real reason that $\text{tanh}$ is preferred compared to $\text{sigmoid}$, especially when it comes to big data when you are usually struggling to find quickly the local (or global) minimum, is that the derivatives of the $\text{tanh}$ are larger than the derivatives of the $\text{sigmoid}$. In other words, you minimize your cost function faster if you use $\text{tanh}$ as an activation fuction.

But why does the hyperbolic tangent have larger derivatives? Just to give you a very simple intuition you may observe the following graph:

The fact that the range is between -1 and 1 compared to 0 and 1, makes the function to be more convenient for neural networks. Apart from that, if I use some math, I can prove that:

$$\tanh{x} = 2σ(2x)-1$$

And in general, we may prove that in most cases $\Big|\frac{\partial\tanh (x)}{\partial x}\Big| > \Big|\frac{\partial\text{σ} (x)}{\partial x}\Big|$.

Answer 3

It all essentially depends on the derivatives of the activation function, the main problem with the sigmoid function is that the max value of its derivative is 0.25, this means that the update of the values of W and b will be small.

The tanh function on the other hand, has a derivativ of up to 1.0, making the updates of W and b much larger.

This makes the tanh function almost always better as an activation function (for hidden layers) rather than the sigmoid function.

To prove this myself (at least in a simple case), I coded a simple neural network and used sigmoid, tanh and relu as activation functions, then I plotted how the error value evolved and this is what I got.

The full notebook I wrote is here https://www.kaggle.com/moriano/a-showcase-of-how-relus-can-speed-up-the-learning

If it helps, here are the charts of the derivatives of the tanh function and the sigmoid one (pay attention to the vertical axis!)

Answer 4

Answering the part of the question so far unaddressed:

Andrew Ng says that using the logistic function (commonly know as sigmoid) really only makes sense in the final layer of a binary classification network.

As the output of the network is expected to be between $0$ and $1$, the logistic is a perfect choice as it's range is exactly $(0, 1)$. No scaling and shifting of $tanh$ required.

Answer 5

And especially when you are solving an Ordinary Differential Equation with a stiff gradient.

tanh(x) converges earlier than sigmoids of all types.

Answer 6

Zero mean assumption relates to the curvature of the error surface.

The whitening argument (https://stats.stackexchange.com/a/330885/27556) is a standard issue with gradient descent. just consider linear regression with gradient descent - the curvature of the error surface is given by the covariance of the inputs (including bias term). [ for a nonlinear model such as a neural net, I would argue you have the same issues]

Zero mean ensures you have no covariance with the bias term ( and the other non zero mean inputs).

If you have covariance with a lot of inputs, then you create a narrow valley shape (in our case across the bias/constant input direction). This is hard for gradient descent to handle (because you would want a bigger learning rate along the valley and a shorter one across the valley), gradient descent has a single learning rate, so you are forced to have a very low learning rate (making slow progress along the valley) to avoid jumping from peak to peak of the valley. Having different learning rates in each direction is essentially what second order methods do, but they are too memory intensive for large parameter models such as neural networks.

Answer 7

Generally, the non-zero centered activation function restricts the movement of parameters over the surface area in some specific directions. Which makes the training slower because it needs mush steps to move from the initial point to the minimum point with these restricted movements. For more details watch only 7 min from this video starting from time 8:43.