14
$\begingroup$

I have read elsewhere that one's choice of hidden layer activation function in a NN should be based on one's need, i.e. if you need values in the range -1 to 1 use tanh and use sigmoid for the range 0 to 1.

My question is how does one know what one's need is? Is it based on the range of the input layer, e.g. use the function that can encompass the input layer's full range of values, or somehow reflects the input layer's distribution (Gaussian function)? Or is the need problem/domain specific and one's experience/judgement is required to make this choice? Or is it simply "use that which gives the best cross-validated minimum training error?"

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ This argument is bs because (tanh+1)/2 is also in 0-1, not to mention that "sigmoid" is such a vague term that it quite often covers tanh. $\endgroup$ – user88 Sep 6 '12 at 11:54
  • $\begingroup$ It's probably worth mentioning that any data set can be normalized to 0->1 and made to use a sigmoid activation 1 + (1 / exp(-sum)). Making the need very difficult to understand without trying both on each data set. The need as you describe it here is tied to the actual relation being learned, ie a binary data set will learn faster or not at all given different activations. $\endgroup$ – Adrian Seeley May 4 '14 at 16:47
12
$\begingroup$

LeCun discusses this in Efficient Backprop Section 4.4. The motivation is similar to the motivation for normalizing the input to zero mean (Section 4.3). The average outputs of the tanh activation function are more likely to be close to zero than the sigmoid, whose average output must be positive.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ A very informative read! $\endgroup$ – babelproofreader Sep 6 '12 at 21:40
6
$\begingroup$

The need mentioned in the first paragraph of the question relates to the output layer activation function, rather than the hidden layer activation function. Having outputs that range from 0 to 1 is convenient as that means they can directly represent probabilities. However, IIRC, a network with tanh output layer activation functions can be trivially transformed into a network with logistic output layer activation function, so it doesn't really matter much in practice.

IIRC the reason for using tanh rather than logistic activation function in the hidden units, which is that change made to a weight using backpropagation depends on both the output of the hidden layer neuron and on the derivative of the activation function, so using the logistic activation function you can have both go to zero at the same time, which can end up with the hidden layer unit becoming frozen.

In short, use tanh for hidden layer activation functions, chose the output layer activation function to enforce desired constraints on the output (common choices: linear - no constraints, logistic - output lies between 0 and 1 and exponential - output strictly positive).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I don't get the "...have both go zero...". I see output might be zero but how is it possible to have logistic function's derivative go zero as tanh not. $\endgroup$ – erogol Nov 1 '12 at 22:10
  • $\begingroup$ it doesn't go exactly to zero, for the logistic function, it just becomes very small. For the tanh function, the derivative is at its largest when the output is zero and the output at its largest when the derivative is smallest. The original paper was written in the late 80s, I'll see if I can remember the details. $\endgroup$ – Dikran Marsupial Nov 2 '12 at 9:53
  • 2
    $\begingroup$ I can't find the original paper, but some of the papers in the book "Neural Networks - Tricks of the Trade" suggest that tanh is better in the hidden layers as networks perform better if the hidden layer activations are centered (i.e. zero mean). $\endgroup$ – Dikran Marsupial Nov 2 '12 at 10:26
2
$\begingroup$

You may use $1.7159 \times \tanh(x \times (2/3))$ on hidden layers. This sigmoid has the property that it has max of its second derivatives at $-1$ and $+1$ values while its asymptotic limits are $[-1.5,+1.5]$. In that way you network will be more accurate on the points near the decision boundary.

The general concept to choose sigmoid for your purpose is to choose the one according to the rule, your output values are in the the range of points, makes the second derivative of sigmoid function maximum.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.