52
$\begingroup$

While the choice of activation functions for the hidden layer is quite clear (mostly sigmoid or tanh), I wonder how to decide on the activation function for the output layer. Common choices are linear functions, sigmoid functions and softmax functions. However, when should I use which one?

$\endgroup$
  • 5
    $\begingroup$ More recently ReLU has become popular as the activation function for hidden units. $\endgroup$ – ijuneja Nov 7 '18 at 14:48
51
$\begingroup$
  • Regression: linear (because values are unbounded)
  • Classification: softmax (simple sigmoid works too but softmax works better)

Use simple sigmoid only if your output admits multiple "true" answers, for instance, a network that checks for the presence of various objects in an image. In other words, the output is not a probability distribution (does not need to sum to 1).

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Softmax is also important if you have multiple groups for classification $\endgroup$ – cdeterman Jun 13 '16 at 12:35
  • 20
    $\begingroup$ I think it's incorrect to say that softmax works "better" than a sigmoid, but you can use softmax in cases in which you cannot use a sigmoid. For binary classification, the logistic function (a sigmoid) and softmax will perform equally well, but the logistic function is mathematically simpler and hence the natural choice. When you have more than two classes, however, you can't use a scalar function like the logistic function as you need more than one output to know the probabilities for all the classes, hence you use softmax. $\endgroup$ – HelloGoodbye Oct 9 '16 at 12:29
  • 2
    $\begingroup$ I don't think softmax is an activation. Generally speaking, you have to activate it before you normalize (softmax) it. $\endgroup$ – Aerin Jul 11 '17 at 16:40
  • $\begingroup$ @Aerin of course softmax is an activation. You work directly without it (if your loss function embeds it and accepts the logits instead), but when your network outputs the probabilities, softmax is an activation function in all its rights. $\endgroup$ – Firebug Apr 3 at 17:41
  • 1
    $\begingroup$ linear for regression is not a default choice. it depends on what is your output. suppose you're forecasting nonnegative quantity or a bounded one, such as the GPA between 0-4 etc. $\endgroup$ – Aksakal Jun 11 at 15:50
34
$\begingroup$

I might be late to the party, but it seems that there are some things that need to be cleared out here.

First of all: the activation function $g(x)$ at the output layer often depends on your cost function. This is done to make the derivative $\frac{\partial C}{\partial z}$ of the cost function $C$ with respect to the inputs $z$ at the last layer easy to compute.

As an example, we could use the mean squared error loss $C(y, g(z)) = \frac{1}{2} (y - g(z))^2$ in a regression setting. By setting $g(x) = x$ (linear activation function), we find for the derivative $$\begin{align*} \frac{\partial C(y,g(z))}{\partial z} & = \frac{\partial C(y, g(z))}{\partial g(z)} \cdot \frac{\partial g(z)}{\partial z} \\ & = \frac{\partial}{\partial g(z)}\left(\frac{1}{2} (y - g(z))^2\right) \cdot \frac{\partial}{\partial z}\left(z\right) \\ & = - (y-g(z)) \cdot 1 \\ & = g(z) - y \end{align*}$$ You get the same, easy expression for $\frac{\partial C}{\partial z}$ if you combine cross-entropy loss with the logistic sigmoid or softmax activation functions.

This is the reason why linear activations are often used for regression and logistic/softmax activations for binary/multi-class classification. However, nothing keeps you from trying out different combinations. Although the expression for $\frac{\partial C}{\partial z}$ will probably not be so nice, it does not imply that your activation function would perform worse.

Second, I would like to add that there are plenty of activation functions that can be used for the hidden layers. Sigmoids (like the logistic function and hyperbolic tangent) have proven to work well indeed, but as indicated by Jatin, these suffer from vanishing gradients when your networks become too deep. In that case ReLUs have become popular. What I would like to emphasise though, is that there are plenty more activation functions available and different researchers keep on looking for new ones (e.g. Exponential Linear Units (ELUs), Gaussian Error Linear Units (GELUs), ...) with different/better properties

To conclude: When looking for the best activation functions, just be creative. Try out different things and see what combinations lead to the best performance.


Addendum: For more pairs of loss functions and activations, you probably want to look for (canonical) link functions

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ why isn't this the best answer ? pretty darned intuitive and completely scientific $\endgroup$ – Vikram Murthy Jun 2 '19 at 12:41
13
$\begingroup$

Sigmoid and tanh should not be used as activation function for the hidden layer. This is because of the vanishing gradient problem, i.e., if your input is on a higher side (where sigmoid goes flat) then the gradient will be near zero. This will cause very slow or no learning during backpropagation as weights will be updated with really small values.

Detailed explanation here: http://cs231n.github.io/neural-networks-1/#actfun

The best function for hidden layers is thus ReLu.

| cite | improve this answer | |
$\endgroup$
  • 15
    $\begingroup$ The question is asking for output layer. -1 $\endgroup$ – Euler_Salter Oct 23 '17 at 14:09
  • 2
    $\begingroup$ Agreed. Added the answer based on the first line of question. Maybe this should have been a comment instead of an answer. $\endgroup$ – Jatin Oct 24 '17 at 21:24
  • 2
    $\begingroup$ Well, but then wouldn't you also get "dead ReLU neurons problem?" Plus, vanishijg gradient problem can be "solved" by batch normalization. If you still want to "deactivate" some neurons the way ReLU activations do, then you can just randomly turn off neurons by drop outs. So i think at the end, it all depends on the problem and just use what works the best $\endgroup$ – Kevvy Kim Oct 23 '18 at 20:25
8
$\begingroup$

Softmax outputs produce a vector that is non-negative and sums to 1. It's useful when you have mutually exclusive categories ("these images only contain cats or dogs, not both"). You can use softmax if you have $2,3,4,5,...$ mutually exclusive labels.

Using $2,3,4,...$ sigmoid outputs produce a vector where each element is a probability. It's useful when you have categories that are not mutually exclusive ("these images can contain cats, dogs, or both cats and dogs together"). You use as many sigmoid neurons as you have categories, and your labels should not be mutually exclusive.

A cute trick is that you can also use a single sigmoid unit if you have a mutually-exclusive binary problem; because a single sigmoid unit can be used to estimate $p(y=1)$, the Kolmogorov axioms imply that when $y$ is binary, we have $1-p(y=1)=p(y=0)$.

Using the identity function as an output can be helpful when your outputs are unbounded. For example, some company's profit or loss for a quarter could be unbounded on either side.

ReLU units or similar variants can be helpful when the output is bounded above (or below, if you reverse the sign). If the output is only restricted to be non-negative, it would make sense to use a ReLU activation as the output function.

Likewise, if the outputs are somehow constrained to lie in $[-1,1]$, tanh could make sense.

The nice thing about neural networks is that they're incredibly flexible tools, and flexibility in output activation is one aspect of that flexibility.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

The choice of the activation function for the output layer depends on the constraints of the problem. I will give my answer based on different examples:

  1. Fitting in Supervised Learning: any activation function can be used in this problem. In some cases, the target data would have to be mapped within the image of the activation function.
  2. Binary decisions: sigmoid or softmax. Examples:
  • Supervised Learning: classification of images in two classes A/B (cats/dogs, number/letter, art/non-art):
    • sigmoid: the output could correspond to the confidence $c$ (valued between 0 and 1) that the image belongs to the first class. The value $1-c$ could be interpreted as the confidence that the image belongs to the second class.
    • softmax: the outputs could be interpreted as the confidences $c_1$ and $c_2$ that the image belongs to each class.
  • Reinforcement Learning: button actions in policy gradient:
    • sigmoid: the output could correspond to the probability $p$ of pressing the button; the probability of not pressing it would be $1-p$.
    • softmax: the outputs would correspond to the probabilities $p_1, p_2$ of pressing/not pressing the button.
  1. Multiple decisions: softmax. Examples:
  • Supervised Learning: classification of images in multiple classes A/B/C/... (for example, classification of digits 0/1/2/3/4/5/6/7/8/9; see MNIST)
    • softmax: in these cases the softmax function is usually chosen so that the sum of all the confidences $\{c_i\}_i$ adds up to 1. The most "reliable" class would have an output closer to $1$.
    • maxout: however, it is also possible to choose the class according to the maximum value of the layer (see this paper).
  • Reinforcement Learning: joestick-type actions in policy gradient; actions that exclude other actions (see openai gym environments; for example LunarLander-v2 where possible actions are "do nothing, fire left orientation engine, fire main engine OR fire right orientation engine."):
    • softmax: the outputs of this function would represent the probabilities $\{p_i\}_i$ of choosing each action.
    • others: they are not usually used since one needs to parameterize the probability of each action. There could be some other way to parameterize the probabilities, but that would probably result in a complicated expression for the computation of the gradient: $$ \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}}( a_k | s_k ) $$
  1. Continuous actions in policy gradient (reinforcement learning); actions that do not take discrete values ​​(see openai gym environments; for example BipedalWalker-v2 where the actions are the amount of torque applied to each joint of the robot): in these cases a probability distribution is usually defined, and used to choose the actions; each action has an associated probability density. The output layers would parameterize the probability distribution. A couple of examples of distributions would be:
  • Normal distribution parametrized by the mean $\mu$ and variance $\sigma^2$: in this case an output layer would provide the mean of the distribution, and another one would provide the variance:
    • if $\mu$ can take values ​​on all $\mathbb{R}$, activation functions like identity, arcsinh, or even lrelu could be used.
    • if $\mu$ can take values ​​in a range $(a, b)$, activation functions such as sigmoid, tanh, or any other whose range is bounded could be used.
    • for $\sigma^2$ it is convenient to use activation functions that produce strictly positive values ​​such as sigmoid, softplus, or relu.
  • Beta distribution parameterized by $a$ and $b$: in this case, $a, b> 0$, so any activation function that produces values ​​greater than $0$ could be convenient. However, this distribution usually presents problems when $a, b <1$, which is why activation functions are often designed to produce values ​​greater than or equal to $1$. A toy example would be softplus+1.

***As a side note on the choice of activation functions in the HIDDEN layers. Only if you are interested in how different activation functions perform, please check the following video:

https://www.youtube.com/watch?v=Hb3vIYUQ_I8

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

The reason why different activations exist is because they match different problems. Among a few others are mentioned by you "linear functions, sigmoid functions and softmax functions":

  • linear is an obvious choice for regression problems where you are predicting unbounded quantities, e.g. stock log returns.
  • sigmoid can be used for regression of bounded quantities, such as probabilities between 0 and 1, and also for classification into two categories such as male/female
  • softmax is used for classification, especially between multiple categories, e.g. activities such as "walking", "sleeping", "running" in activity trackers. I had a curious case recently receiving an email from a local association which started with something like "based on your profile answers we think you are white...", I wonder what was their ML algo
| cite | improve this answer | |
$\endgroup$
-1
$\begingroup$

As i see that lot of people have given very intuitive, and perfect answers for the question, but i will give my answer in a nutshell.

So, For hidden layers the best option to use is ReLU, and the second option you can use as SIGMOID. For output layers the best option depends, so we use LINEAR FUNCTIONS for regression type of output layers and SOFTMAX for multi-class classification.

I just gave one method for each type of classification to avoid the confusion, and also you can try other functions also to get better understanding.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.