119
$\begingroup$

The tanh activation function is:

$$tanh \left( x \right) = 2 \cdot \sigma \left( 2 x \right) - 1$$

Where $\sigma(x)$, the sigmoid function, is defined as:
$$\sigma(x) = \frac{e^x}{1 + e^x}$$.

Questions:

  • Does it really matter between using those two activation functions (tanh vs. sigma)?
  • Which function is better in which cases?
Improve this question
$\endgroup$
4
  • 18
    $\begingroup$ $\textrm{tanh}(x) = 2\sigma(2x) - 1$ $\endgroup$
    – Roman Shapovalov
    Sep 23, 2014 at 15:13
  • $\begingroup$ Deep Neural Networks have moved on. The current preference is the RELU function. $\endgroup$
    – Paul Nord
    Oct 4, 2017 at 4:35
  • 15
    $\begingroup$ @PaulNord Both tanh and sigmoids are still used in conjunction with other activations like RELU, depends what you're trying to do. $\endgroup$
    – Tahlor
    Oct 27, 2017 at 17:36
  • $\begingroup$ @RomanShapovalov so we can say that having 2 layers sigmoid + linear are at least as powerful as a layer with tanh? $\endgroup$
    – Alberto Sinigaglia
    Jul 31, 2022 at 17:35

2 Answers 2

Reset to default
114
$\begingroup$

Yes it matters for technical reasons. Basically for optimization. It is worth reading Efficient Backprop by LeCun et al.

There are two reasons for that choice (assuming you have normalized your data, and this is very important):

  1. Having stronger gradients: since data is centered around 0, the derivatives are higher. To see this, calculate the derivative of the tanh function and notice that its range (output values) is [0,1].

The range of the tanh function is [-1,1] and that of the sigmoid function is [0,1]

  1. Avoiding bias in the gradients. This is explained very well in the paper, and it is worth reading it to understand these issues.
Improve this answer
$\endgroup$
16
  • $\begingroup$ I have small doubt in the paper you suggested. In page 14, "When MLP have shared weights (eg:Convolutional nets), Learning rate should be choosen in such a way that, it is proportional to square root of no. of connections sharing the weight." Can you please explain why? $\endgroup$
    – satya
    Jun 9, 2014 at 13:06
  • $\begingroup$ this question has already been answered here stats.stackexchange.com/questions/47590/… $\endgroup$
    – jpmuc
    Jun 9, 2014 at 13:55
  • 1
    $\begingroup$ That is a very general question. Long story short: the cost function determines what the neural network should do: classification or regression and how. If you could get a copy of "Neural Networks for Pattern Recognition" by Christopher Bishop that'd be great. Also "Machine Learning" by Mitchell gives you a good explanation at a more basic level. $\endgroup$
    – jpmuc
    Jun 10, 2014 at 9:30
  • 1
    $\begingroup$ I am sorry, Satya, I am usually quite busy during the week. How do you normalize your data exactly? en.wikipedia.org/wiki/Whitening_transformation I am not really sure what your problem can be. The easiest way is to substract the mean and then equalize with the covariance matrix. Evtl. you need to add some component for high frequencies (see ZCA transform in the reference above) $\endgroup$
    – jpmuc
    Jun 11, 2014 at 21:27
  • 1
    $\begingroup$ Thanks a lot juampa. You are really helping me a lot. Suggested reading are very good. I am actually doing a project on climate data mining. 50% of my input features are temperature(range 200K-310K) and 50% of my input features are pressure values(range 50000pa to 100000pa). I am doing whitening. Before pca, is there any need to normalize it... If yes, how should I normalize it? Should I normalize before subtracting by mean or after subtracting by mean? I am getting different results if I am normalizing by different methods... $\endgroup$
    – satya
    Jun 12, 2014 at 10:22
49
$\begingroup$

Thanks a lot @jpmuc ! Inspired by your answer, I calculated and plotted the derivative of the tanh function and the standard sigmoid function seperately. I'd like to share with you all. Here is what I got. This is the derivative of the tanh function. For input between [-1,1], we have derivative between [0.42, 1]. enter image description here

This is the derivative of the standard sigmoid function f(x)= 1/(1+exp(-x)). For input between [0,1], we have derivative between [0.20, 0.25]. enter image description here

Apparently the tanh function provides stronger gradients.

Improve this answer
$\endgroup$
6
  • 8
    $\begingroup$ Another way of looking at this is that σ(2x) is the same as σ(x), but with a horizontal stretch applied, scale factor 1/2 (i.e. it's the same graph but with everything squashed in towards the y axis). When you squash it in, the slope gets steeper $\endgroup$
    – rbennett485
    Mar 3, 2017 at 16:05
  • 3
    $\begingroup$ I don't see why this would make any difference. The scale and squashing will be random for each node, and (with offsets and weights on input and output) both will be universal approximators, converging to the same result. $\endgroup$
    – endolith
    Nov 25, 2018 at 16:31
  • 2
    $\begingroup$ @endolith The question is not one of efficacy, but of efficiency. A model that converges to the same result but takes a hundred times as long to do so is not a model I want to be using $\endgroup$
    – TheEnvironmentalist
    Feb 10, 2020 at 6:28
  • 1
    $\begingroup$ @TheEnvironmentalist I mean that with bias, they're exactly the same function, so efficacy and efficiency are the same. $\endgroup$
    – endolith
    Feb 10, 2020 at 17:30
  • 2
    $\begingroup$ Depends on how you're using these activation functions, but you can just increase the learning rate, or multiply loss/layer outputs by some factor, and then convergence will be similar. So in that respect you are correct @endolith $\endgroup$
    – Azmisov
    Aug 8, 2020 at 1:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.