12
$\begingroup$

In the machine learning literature, to represent a probability distribution, the softmax function is often used. Is there a reason for this? Why isn't another function used?

$\endgroup$

4 Answers 4

11
$\begingroup$

From an optimization perspective it has some nice properties in terms of differentiability. For a lot of machine learning problems it's a good fit for 1-of-N classification.

From a deep learning perspective: One could also argue that in theory, using a deep network with a softmax classifier on top can represent any N-class probability function over the feature space as MLPs have the Universal Approximation property.

$\endgroup$
3
  • 2
    $\begingroup$ So the main reason for popularity of Softmax is it's nice differentiation properties which are helpful in Gradient Based learning setting. That's it, right? $\endgroup$ Jan 6, 2016 at 18:42
  • $\begingroup$ Yeap, in my opinion anyway. Softmax is a simple with nice derivatives and is attractive for gradient based learning. Agree with everything you said. $\endgroup$
    – Indie AI
    Jan 6, 2016 at 18:46
  • $\begingroup$ You can think softmax as probability mass/density function of the function you are going to optimize. In my opinion, softmax is just a convenience way to model a probability mass/density function. $\endgroup$ Dec 22, 2018 at 21:55
5
$\begingroup$

Softmax is also a generalization of the logistic sigmoid function and therefore it carries the properties of the sigmoid such as ease of differentiation and being in the range 0-1. The output of a logistic sigmoid function is also between 0 and 1 and therefore naturally a suitable choice for representing probability. Its derivative is also exoressed in terms of its own output. However, if your function has a vector output you need to use the Softmax function to get the probability distribution over the output vector. There are some other advantages of using Softmax which Indie AI has mentioned, although it does not necessarily has anything to do with the Universal Approximation theory since Softmax is not a function only used for Neural Networks.

References

$\endgroup$
5
$\begingroup$

See section 4.2 of Bishop's Pattern Recognition and Machine Learning on probabilistic generative models. He shows on page 197 that in the two-class case, given classes $ \mathcal{C}_1, \mathcal{C}_2 $, that $ p(\mathcal{C}_1 \mid \mathbf{x}) $, the posterior for class $ \mathcal{C}_1 $ (i.e. conditional on input example $ \mathbf{x} \in \mathbb{R}^d $), is such that (by Bayes' rule)

$$ p(\mathcal{C}_1 \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \mathcal{C}_1)p(\mathcal{C}_1)}{p(\mathbf{x} \mid \mathcal{C}_1)p(\mathcal{C}_1) + p(\mathbf{x} \mid \mathcal{C}_2)p(\mathcal{C}_2)} = \frac{1}{1 + e^{-a(\mathbf{x})}} = \sigma \circ a(\mathbf{x}) $$

Here $ \sigma : \mathbb{R} \rightarrow (0, 1) $ is the sigmoid function and $ a : \mathbb{R}^d \rightarrow \mathbb{R} $ is such that

$$ a(\mathbf{x}) \triangleq \log\frac{p(\mathbf{x} \mid\mathcal{C}_1)p(\mathcal{C}_1)}{p(\mathbf{x} \mid \mathcal{C}_2)p(\mathcal{C}_2)} = \log\frac{p(\mathcal{C_1}, \mathbf{x})}{p(\mathcal{C}_2, \mathbf{x})} $$

In the multiclass case, i.e. with classes $ \mathcal{C}_1, \ldots \mathcal{C}_K $, we naturally have for $ k \in \{1, \ldots K\} $,

$$ p(\mathcal{C}_k \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid \mathcal{C}_k)p(\mathcal{C}_k)}{\sum_{j = 1}^Kp(\mathbf{x} \mid \mathcal{C}_k)p(\mathcal{C}_j)} = \frac{e^{a_k(\mathbf{x})}}{\sum_{j = 1}^Ke^{a_j(\mathbf{x})}} $$

Here for $ k \in \{1, \ldots K\} $, $ a_k : \mathbb{R}^d \rightarrow \mathbb{R} $ is such that

$$ a_k(\mathbf{x}) \triangleq \log\left(p(\mathbf{x} \mid \mathcal{C}_k)p(\mathcal{C}_k)\right) = \log p(\mathcal{C}_k, \mathbf{x}) $$

The $ a, a_k $ functions can be given parametric forms--for example, in multiclass logistic regression, $ a_k(\mathbf{x}) \triangleq \mathbf{w}_k^\top\mathbf{x} + b_k $ for $ \mathbf{w}_k \in \mathbb{R}^d, b_k \in \mathbb{R} $. In fact, page 203 states that for class conditional distributions, i.e. $ X \mid \mathcal{C}_k $, that are members of the exponential family of distributions, the $ a_k $ functions are affine functions of $ \mathbf{x} $. An example is linear discriminant analysis, which assumes Gaussian class-conditional distributions with a shared covariance matrix, as equation (4.68) on page 199 shows that the $ a_k $ function is affine.

The softmax function itself, probabilistic interpretations aside, is a smooth, differentiable approximation to the max function, which of course the other answers have mentioned is helpful when using gradient-based methods to minimize an objective function. For example, the binary (multiclass) logistic regression objective is convex and differentiable, with the differentiability partly because of its inclusion of the sigmoid (softmax) function.

$\endgroup$
5
$\begingroup$

The softmax function has a number of desirable properties for optimisation and other mathematical methods dealing with probability vectors. Its most important property is that it gives a mapping that allows you to represent any probability vector as a point in unconstrained Euclidean space, but it does this in a way that has some nice smoothness properties and other properties that are useful in various types of problems.

Given any scale parameter $\lambda>0$ the probability vector $\mathbf{p} = (p_0,p_1,...,p_n)$ can be mapped to or from a point $\boldsymbol{\eta} \in \mathbb{R}^n$ using the softmax function and inverse-softmax function:

$$\begin{align} \text{soft}(\boldsymbol{\eta}) &= \Bigg( \frac{1}{1 + \sum_{i=1}^n \exp(\lambda \eta_i)}, \frac{\exp(\lambda \eta_1)}{1 + \sum_{i=1}^n \exp(\lambda \eta_i)}, ..., \frac{\exp(\lambda \eta_n)}{1 + \sum_{i=1}^n \exp(\lambda \eta_i)} \Bigg), \\[12pt] \text{invsoft}(\mathbf{p}) &= \Bigg( \frac{\log(p_1) - \log(p_0)}{\lambda}, ..., \frac{\log(p_n) - \log(p_0)}{\lambda} \Bigg). \\[12pt] \end{align}$$

This mapping is sufficient for all probability vectors with non-zero elements, which also gets you arbitrarily close to any probability vector with one or more zero elements. The mapping has several useful properties:

  • The domain of the softmax function is unconstrained Euclidean space, which makes it useful in optimisation problems that have a probability vector as an input. Specifically, if you have an objective function $H$ that maps an input probability vector to a real number, you can form the function composition $G = H \circ s : \mathbb{R}^n \rightarrow \mathbb{R}$ to convert the problem to an unconstrained optimisation.

  • The softmax function is an analytic function (i.e., it is infinitely differentiable and has a convergent Taylor series) which means it is nice and smooth and can be represented closely by a polynomial in a neighbourhood of any point.

  • The first two derivatives of the softmax function and inverse-softmax function have simple forms and can be computed explicitly. This is also useful for optimisation problems and other mathematical problems involving probability vectors.

Of course, it is possible to form other mappings from unconstrained Euclidean space to the space of probability vectors, and other forms might also be similarly useful for some purposes. The softmax function is a form that has simple derivatives and so it is useful in a range of optimisation problems and other mathematical methods involving probability vectors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.