122
$\begingroup$

The tanh activation function is:

$$\tanh x= 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:

  • Does it really matter between using those two activation functions (tanh vs. sigma)?
  • Which function is better in which cases?
Improve this question
$\endgroup$
4
  • 19
    $\begingroup$ $\textrm{tanh}(x) = 2\sigma(2x) - 1$ $\endgroup$
    – Roman Shapovalov
    Commented Sep 23, 2014 at 15:13
  • $\begingroup$ Deep Neural Networks have moved on. The current preference is the RELU function. $\endgroup$
    – Paul Nord
    Commented Oct 4, 2017 at 4:35
  • 15
    $\begingroup$ @PaulNord Both tanh and sigmoids are still used in conjunction with other activations like RELU, depends what you're trying to do. $\endgroup$
    – Tahlor
    Commented Oct 27, 2017 at 17:36
  • $\begingroup$ @RomanShapovalov so we can say that having 2 layers sigmoid + linear are at least as powerful as a layer with tanh? $\endgroup$
    – Alberto
    Commented Jul 31, 2022 at 17:35

3 Answers 3

Reset to default
119
$\begingroup$

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):

  1. Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]

  1. Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
Improve this answer
$\endgroup$
16
  • $\begingroup$ I have small doubt in the paper you suggested. In page 14, "When MLP have shared weights (eg:Convolutional nets), Learning rate should be choosen in such a way that, it is proportional to square root of no. of connections sharing the weight." Can you please explain why? $\endgroup$
    – satya
    Commented Jun 9, 2014 at 13:06
  • $\begingroup$ this question has already been answered here stats.stackexchange.com/questions/47590/… $\endgroup$
    – jpmuc
    Commented Jun 9, 2014 at 13:55
  • 1
    $\begingroup$ That is a very general question. Long story short: the cost function determines what the neural network should do: classification or regression and how. If you could get a copy of "Neural Networks for Pattern Recognition" by Christopher Bishop that'd be great. Also "Machine Learning" by Mitchell gives you a good explanation at a more basic level. $\endgroup$
    – jpmuc
    Commented Jun 10, 2014 at 9:30
  • 1
    $\begingroup$ I am sorry, Satya, I am usually quite busy during the week. How do you normalize your data exactly? en.wikipedia.org/wiki/Whitening_transformation I am not really sure what your problem can be. The easiest way is to substract the mean and then equalize with the covariance matrix. Evtl. you need to add some component for high frequencies (see ZCA transform in the reference above) $\endgroup$
    – jpmuc
    Commented Jun 11, 2014 at 21:27
  • 1
    $\begingroup$ Thanks a lot juampa. You are really helping me a lot. Suggested reading are very good. I am actually doing a project on climate data mining. 50% of my input features are temperature(range 200K-310K) and 50% of my input features are pressure values(range 50000pa to 100000pa). I am doing whitening. Before pca, is there any need to normalize it... If yes, how should I normalize it? Should I normalize before subtracting by mean or after subtracting by mean? I am getting different results if I am normalizing by different methods... $\endgroup$
    – satya
    Commented Jun 12, 2014 at 10:22
50
$\begingroup$

Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got. This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1]. enter image description here

This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)). For input between [0,1], we have derivative between [0.20, 0.25]. enter image description here

Apparently the tanh function provides stronger gradients.

Improve this answer
$\endgroup$
6
  • 8
    $\begingroup$ Another way of looking at this is that σ(2x) is the same as σ(x), but with a horizontal stretch applied, scale factor 1/2 (i.e. it's the same graph but with everything squashed in towards the y axis). When you squash it in, the slope gets steeper $\endgroup$
    – rbennett485
    Commented Mar 3, 2017 at 16:05
  • 3
    $\begingroup$ I don't see why this would make any difference. The scale and squashing will be random for each node, and (with offsets and weights on input and output) both will be universal approximators, converging to the same result. $\endgroup$
    – endolith
    Commented Nov 25, 2018 at 16:31
  • 2
    $\begingroup$ @endolith The question is not one of efficacy, but of efficiency. A model that converges to the same result but takes a hundred times as long to do so is not a model I want to be using $\endgroup$
    – TheEnvironmentalist
    Commented Feb 10, 2020 at 6:28
  • 1
    $\begingroup$ @TheEnvironmentalist I mean that with bias, they're exactly the same function, so efficacy and efficiency are the same. $\endgroup$
    – endolith
    Commented Feb 10, 2020 at 17:30
  • 2
    $\begingroup$ Depends on how you're using these activation functions, but you can just increase the learning rate, or multiply loss/layer outputs by some factor, and then convergence will be similar. So in that respect you are correct @endolith $\endgroup$
    – Azmisov
    Commented Aug 8, 2020 at 1:54
0
$\begingroup$

Generally speaking, $\tanh$ has two main advantages over a sigmoid function:

  1. It has a slightly bigger derivative than the sigmoid (at least for the area around 0), which helps it to cope a bit better with the “vanishing gradients” problem of deep neural networks. Here is a plot of the derivatives of both functions:

enter image description here

  1. It is symmetric around 0, which helps it to avoid the “bias shift” problem that sigmoid suffer from (which causes the weight vectors to move in diagonals, or “zig-zag”, which slows down learning).

Sigmoid has 1 main advantage over $\tanh$, which is that it can represent a binary probability - hence can be used as the output of the final layer in binary classification problems.

You can check out this video I made on YouTube which explains a bit further about these problems.

Elaboration on the bias shift problem:

Consider a case of activation functions like Sigmoids which only output positive values. Now let’s focus on a single layer $a_l$. Let’s look at the weight vector associated with the first next neuron: $z_{(l+1),1}=W_{l,1}\cdot a_l + b_{l,1}$.

enter image description here

The gradient w.r.t. this vector will be (according to the chain rule) $a_l \cdot \frac{\partial \mathcal L}{\partial z_{(l+1),1}}$. That is the gradient up to $z_{(l+1),1}$ (which is a scalar) times the gradient of $z_{(l+1),1}$ w.r.t. $W_{l1}$ which is just $a_l$.

We know that the $a_l$ neurons are all $\ge 0$, so this $W_{l1}$ vector updates depend on $sign(\frac{\partial \mathcal L}{\partial z_{(l+1),1}}).$

This means that the vector either increase or decrease for all elements $\Rightarrow$ it can only move in Zig-Zag / diagonals, which is not very efficient. This is sometimes called the “bias shift” problem. It also happens when the activations output values which are far from 0 (though to a less extent).

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.