While the choice of activation functions for the hidden layer is quite clear (mostly sigmoid or tanh), I wonder how to decide on the activation function for the output layer. Common choices are linear functions, sigmoid functions and softmax functions. However, when should I use which one?

  • 4
    $\begingroup$ More recently ReLU has become popular as the activation function for hidden units. $\endgroup$ – ijuneja Nov 7 '18 at 14:48
  • Regression: linear (because values are unbounded)
  • Classification: softmax (simple sigmoid works too but softmax works better)

Use simple sigmoid only if your output admits multiple "true" answers, for instance, a network that checks for the presence of various objects in an image. In other words, the output is not a probability distribution (does not need to sum to 1).

  • 2
    $\begingroup$ Softmax is also important if you have multiple groups for classification $\endgroup$ – cdeterman Jun 13 '16 at 12:35
  • 16
    $\begingroup$ I think it's incorrect to say that softmax works "better" than a sigmoid, but you can use softmax in cases in which you cannot use a sigmoid. For binary classification, the logistic function (a sigmoid) and softmax will perform equally well, but the logistic function is mathematically simpler and hence the natural choice. When you have more than two classes, however, you can't use a scalar function like the logistic function as you need more than one output to know the probabilities for all the classes, hence you use softmax. $\endgroup$ – HelloGoodbye Oct 9 '16 at 12:29
  • 1
    $\begingroup$ I don't think softmax is an activation. Generally speaking, you have to activate it before you normalize (softmax) it. $\endgroup$ – Aerin Jul 11 '17 at 16:40

I might be late to the party, but it seems that there are some things that need to be cleared out here.

First of all: the activation function $g(x)$ at the output layer often depends on your cost function. This is done to make the derivative $\frac{\partial C}{\partial z}$ of the cost function $C$ with respect to the inputs $z$ at the last layer easy to compute.

As an example, we could use the mean squared error loss $C(y, g(z)) = \frac{1}{2} (y - g(z))^2$ in a regression setting. By setting $g(x) = x$ (linear activation function), we find for the derivative $$\begin{align*} \frac{\partial C(y,g(z))}{\partial z} & = \frac{\partial C(y, g(z))}{\partial g(z)} \cdot \frac{\partial g(z)}{\partial z} \\ & = \frac{\partial}{\partial g(z)}\left(\frac{1}{2} (y - g(z))^2\right) \cdot \frac{\partial}{\partial z}\left(z\right) \\ & = - (y-g(z)) \cdot 1 \\ & = g(z) - y \end{align*}$$ You get the same, easy expression for $\frac{\partial C}{\partial z}$ if you combine cross-entropy loss with the logistic sigmoid or softmax activation functions.

This is the reason why linear activations are often used for regression and logistic/softmax activations for binary/multi-class classification. However, nothing keeps you from trying out different combinations. Although the expression for $\frac{\partial C}{\partial z}$ will probably not be so nice, it does not imply that your activation function would perform worse.

Second, I would like to add that there are plenty of activation functions that can be used for the hidden layers. Sigmoids (like the logistic function and hyperbolic tangent) have proven to work well indeed, but as indicated by Jatin, these suffer from vanishing gradients when your networks become too deep. In that case ReLUs have become popular. What I would like to emphasise though, is that there are plenty more activation functions available and different researchers keep on looking for new ones (e.g. Exponential Linear Units (ELUs), Gaussian Error Linear Units (GELUs), ...) with different/better properties

To conclude: When looking for the best activation functions, just be creative. Try out different things and see what combinations lead to the best performance.

Addendum: For more pairs of loss functions and activations, you probably want to look for (canonical) link functions

  • 1
    $\begingroup$ why isn't this the best answer ? pretty darned intuitive and completely scientific $\endgroup$ – Vikram Murthy Jun 2 '19 at 12:41

Sigmoid and tanh should not be used as activation function for the hidden layer. This is because of the vanishing gradient problem, i.e., if your input is on a higher side (where sigmoid goes flat) then the gradient will be near zero. This will cause very slow or no learning during backpropagation as weights will be updated with really small values.

Detailed explanation here: http://cs231n.github.io/neural-networks-1/#actfun

The best function for hidden layers is thus ReLu.

  • 13
    $\begingroup$ The question is asking for output layer. -1 $\endgroup$ – Euler_Salter Oct 23 '17 at 14:09
  • 1
    $\begingroup$ Agreed. Added the answer based on the first line of question. Maybe this should have been a comment instead of an answer. $\endgroup$ – Jatin Oct 24 '17 at 21:24
  • $\begingroup$ Well, but then wouldn't you also get "dead ReLU neurons problem?" Plus, vanishijg gradient problem can be "solved" by batch normalization. If you still want to "deactivate" some neurons the way ReLU activations do, then you can just randomly turn off neurons by drop outs. So i think at the end, it all depends on the problem and just use what works the best $\endgroup$ – Kevvy Kim Oct 23 '18 at 20:25

Softmax outputs produce a vector that is non-negative and sums to 1. It's useful when you have mutually exclusive categories ("these images only contain cats or dogs, not both"). You can use softmax if you have $2,3,4,5,...$ mutually exclusive labels.

Using $2,3,4,...$ sigmoid outputs produce a vector where each element is a probability. It's useful when you have categories that are not mutually exclusive ("these images can contain cats, dogs, or both cats and dogs together"). You use as many sigmoid neurons as you have categories, and your labels should not be mutually exclusive.

The a cute trick is that you can also use a single sigmoid unit if you have a mutually-exclusive binary problem; because a single sigmoid unit can be used to estimate $p(y=1)$, the Kolmogorov axioms imply that $1-p(y=1)=p(y=0)$.

Using the identity function as an output can be helpful when your outputs are unbounded. Some company's profit or loss for a quarter could be unbounded on either side.

ReLU units or similar variants can be helpful when the output is bounded above or below. If the output is only restricted to be non-negative, it would make sense to use a ReLU activation as the output function.

Likewise, if the outputs are somehow constrained to lie in $[-1,1]$, tanh could make sense.

The nice thing about neural networks is that they're incredibly flexible tools.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.