Which activation function for output layer?

Question

While the choice of activation functions for the hidden layer is quite clear (mostly sigmoid or tanh), I wonder how to decide on the activation function for the output layer. Common choices are linear functions, sigmoid functions and softmax functions. However, when should I use which one?

Answer 1

• Regression: linear (because values are unbounded)
• Classification: softmax (simple sigmoid works too but softmax works better)

Use simple sigmoid only if your output admits multiple "true" answers, for instance, a network that checks for the presence of various objects in an image. In other words, the output is not a probability distribution (does not need to sum to 1).

Answer 2

I might be late to the party, but it seems that there are some things that need to be cleared out here.

First of all: the activation function $g(x)$ at the output layer often depends on your cost function. This is done to make the derivative $\frac{\partial C}{\partial z}$ of the cost function $C$ with respect to the inputs $z$ at the last layer easy to compute.

As an example, we could use the mean squared error loss $C(y, g(z)) = \frac{1}{2} (y - g(z))^2$ in a regression setting. By setting $g(x) = x$ (linear activation function), we find for the derivative $$\begin{align*} \frac{\partial C(y,g(z))}{\partial z} & = \frac{\partial C(y, g(z))}{\partial g(z)} \cdot \frac{\partial g(z)}{\partial z} \\ & = \frac{\partial}{\partial g(z)}\left(\frac{1}{2} (y - g(z))^2\right) \cdot \frac{\partial}{\partial z}\left(z\right) \\ & = - (y-g(z)) \cdot 1 \\ & = g(z) - y \end{align*}$$ You get the same, easy expression for $\frac{\partial C}{\partial z}$ if you combine cross-entropy loss with the logistic sigmoid or softmax activation functions.

This is the reason why linear activations are often used for regression and logistic/softmax activations for binary/multi-class classification. However, nothing keeps you from trying out different combinations. Although the expression for $\frac{\partial C}{\partial z}$ will probably not be so nice, it does not imply that your activation function would perform worse.

Second, I would like to add that there are plenty of activation functions that can be used for the hidden layers. Sigmoids (like the logistic function and hyperbolic tangent) have proven to work well indeed, but as indicated by Jatin, these suffer from vanishing gradients when your networks become too deep. In that case ReLUs have become popular. What I would like to emphasise though, is that there are plenty more activation functions available and different researchers keep on looking for new ones (e.g. Exponential Linear Units (ELUs), Gaussian Error Linear Units (GELUs), ...) with different/better properties

To conclude: When looking for the best activation functions, just be creative. Try out different things and see what combinations lead to the best performance.

---

Addendum: For more pairs of loss functions and activations, you probably want to look for (canonical) link functions

Answer 3

Sigmoid and tanh should not be used as activation function for the hidden layer. This is because of the vanishing gradient problem, i.e., if your input is on a higher side (where sigmoid goes flat) then the gradient will be near zero. This will cause very slow or no learning during backpropagation as weights will be updated with really small values.

Detailed explanation here: http://cs231n.github.io/neural-networks-1/#actfun

The best function for hidden layers is thus ReLu.

Answer 4

Softmax outputs produce a vector that is non-negative and sums to 1. It's useful when you have mutually exclusive categories ("these images only contain cats or dogs, not both"). You can use softmax if you have $2,3,4,5,...$ mutually exclusive labels.

Using $2,3,4,...$ sigmoid outputs produce a vector where each element is a probability. It's useful when you have categories that are not mutually exclusive ("these images can contain cats, dogs, or both cats and dogs together, or neither cats nor dogs"). You use as many sigmoid neurons as you have categories, and your labels should not be mutually exclusive.

A cute trick is that you can also use a single sigmoid unit if you have a mutually-exclusive binary problem; because a single sigmoid unit can be used to estimate $p(y=1)$, the Kolmogorov axioms imply that when $y$ is binary, we have $1-p(y=1)=p(y=0)$.

Using the identity function as an output can be helpful when your outputs are unbounded. For example, some company's profit or loss for a quarter could be unbounded on either side.

ReLU units or similar variants can be helpful when the output is bounded above (or below, if you reverse the sign). If the output is only restricted to be non-negative, it would make sense to use a ReLU activation as the output function.

Likewise, if the outputs are somehow constrained to lie in $[-1,1]$, tanh could make sense.

The nice thing about neural networks is that they're incredibly flexible tools, and flexibility in output activation is one aspect of that flexibility.

Answer 5

The choice of the activation function for the output layer depends on the constraints of the problem. I will give my answer based on different examples:

0. Fitting in Supervised Learning: any activation function can be used in this problem. In some cases, the target data would have to be mapped within the image of the activation function.
1. Binary decisions: sigmoid or softmax. Examples:

• Supervised Learning: classification of images in two classes A/B (cats/dogs, number/letter, art/non-art):
  • sigmoid: the output could correspond to the confidence $c$ (valued between 0 and 1) that the image belongs to the first class. The value $1-c$ could be interpreted as the confidence that the image belongs to the second class.
  • softmax: the outputs could be interpreted as the confidences $c_1$ and $c_2$ that the image belongs to each class.
• Reinforcement Learning: button actions in policy gradient:
  • sigmoid: the output could correspond to the probability $p$ of pressing the button; the probability of not pressing it would be $1-p$.
  • softmax: the outputs would correspond to the probabilities $p_1, p_2$ of pressing/not pressing the button.

2. Multiple decisions: softmax. Examples:

• Supervised Learning: classification of images in multiple classes A/B/C/... (for example, classification of digits 0/1/2/3/4/5/6/7/8/9; see MNIST)
  • softmax: in these cases the softmax function is usually chosen so that the sum of all the confidences $\{c_i\}_i$ adds up to 1. The most "reliable" class would have an output closer to $1$.
  • maxout: however, it is also possible to choose the class according to the maximum value of the layer (see this paper).
• Reinforcement Learning: joestick-type actions in policy gradient; actions that exclude other actions (see openai gym environments; for example LunarLander-v2 where possible actions are "do nothing, fire left orientation engine, fire main engine OR fire right orientation engine."):
  • softmax: the outputs of this function would represent the probabilities $\{p_i\}_i$ of choosing each action.
  • others: they are not usually used since one needs to parameterize the probability of each action. There could be some other way to parameterize the probabilities, but that would probably result in a complicated expression for the computation of the gradient: $$ \nabla_{\boldsymbol{\theta}} \log \pi_{\boldsymbol{\theta}}( a_k | s_k ) $$

3. Continuous actions in policy gradient (reinforcement learning); actions that do not take discrete values ​​(see openai gym environments; for example BipedalWalker-v2 where the actions are the amount of torque applied to each joint of the robot): in these cases a probability distribution is usually defined, and used to choose the actions; each action has an associated probability density. The output layers would parameterize the probability distribution. A couple of examples of distributions would be:

• Normal distribution parametrized by the mean $\mu$ and variance $\sigma^2$: in this case an output layer would provide the mean of the distribution, and another one would provide the variance:
  • if $\mu$ can take values ​​on all $\mathbb{R}$, activation functions like identity, arcsinh, or even lrelu could be used.
  • if $\mu$ can take values ​​in a range $(a, b)$, activation functions such as sigmoid, tanh, or any other whose range is bounded could be used.
  • for $\sigma^2$ it is convenient to use activation functions that produce strictly positive values ​​such as sigmoid, softplus, or relu.
• Beta distribution parameterized by $a$ and $b$: in this case, $a, b> 0$, so any activation function that produces values ​​greater than $0$ could be convenient. However, this distribution usually presents problems when $a, b <1$, which is why activation functions are often designed to produce values ​​greater than or equal to $1$. A toy example would be softplus+1.

***As a side note on the choice of activation functions in the HIDDEN layers. Only if you are interested in how different activation functions perform, please check the following video:

https://www.youtube.com/watch?v=Hb3vIYUQ_I8

Answer 6

The reason why different activations exist is because they match different problems. Among a few others are mentioned by you "linear functions, sigmoid functions and softmax functions":

• linear is an obvious choice for regression problems where you are predicting unbounded quantities, e.g. stock log returns.
• sigmoid can be used for regression of bounded quantities, such as probabilities between 0 and 1, and also for classification into two categories such as male/female
• softmax is used for classification, especially between multiple categories, e.g. activities such as "walking", "sleeping", "running" in activity trackers. I had a curious case recently receiving an email from a local association which started with something like "based on your profile answers we think you are white...", I wonder what was their ML algo

Answer 7

As i see that lot of people have given very intuitive, and perfect answers for the question, but i will give my answer in a nutshell.

So, For hidden layers the best option to use is ReLU, and the second option you can use as SIGMOID. For output layers the best option depends, so we use LINEAR FUNCTIONS for regression type of output layers and SOFTMAX for multi-class classification.

I just gave one method for each type of classification to avoid the confusion, and also you can try other functions also to get better understanding.