How is softmax unit derived and what is the implication?

Question

I'm trying to understand why the softmax function is defined as such:

$\frac{e^{z_{j}}} {\Sigma^{K}_{k=1}{e^{z_{k}}}} = \sigma(z)$

I understand how this normalizes the data and properly maps to some range (0, 1) but the different between weight probabilities varies exponentially rather than linearly. Is there a reason why we want this behaviour?

Also this equation seems rather arbitrary and I feel that it a large family of equations could satisfy our requirements. I have not seen any derivations online so I'm assuming it is merely a definition. Why not choose any other definition that satisfies the same requirements?

Answer 1

The categorical distribution is the minimum assumptive distribution over the support of "a finite set of mutually exclusive outcomes" given the sufficient statistic of "which outcome happened". In other words, using any other distribution would be an additional assumption. Without any prior knowledge, you must assume a categorical distribution for this support and sufficient statistic. It is an exponential family. (All minimum assumptive distributions for a given support and sufficient statistic are exponential families.)

The correct way to combine two beliefs based on independent information is the pointwise product of densities making sure not to double-count prior information that's in both beliefs. For an exponential family, this combination is addition of natural parameters.

The expectation parameters are the expected values of $x_k$ where $x_k$ are the number of times you observed outcome $k$. This is the right parametrization for converting a set of observations to a maximum likelihood distribution. You simply average in this space. This is what you want when you are modeling observations.

The multinomial logistic function is the conversion from natural parameters to expectation parameters of the categorical distribution. You can derive this conversion as the gradient of the log-normalizer with respect to natural parameters.

In summary, the multinomial logistic function falls out of three assumptions: a support, a sufficient statistic, and a model whose belief is a combination of independent pieces of information.

Answer 2

I know this is a late post, but I do feel like there would still be some value in providing some justification for those who happen to land here.

You're not entirely wrong. It is arbitrary to a certain extent, but perhaps arbitrary is the wrong word. It is more like a design choice. Let me explain.

It turns out that the Softmax is actually the generalization of the Sigmoid function, which is a Bernoulli (output 0 or 1) output unit:

$ \begin{equation} [1+\text{exp}(-z)]^{-1} \end{equation} $

But where does the Sigmoid function come from, you might ask.

Well, it turns out that many different probability distributions including the Bernoulli, Poisson distribution, Gaussian, etc follow something called a Generalized Linear Model (GLM). That is, they may be expressed in terms of:

$ \begin{equation} P(y;\eta) = b(y)\text{exp}[\eta^TT(y) - a(\eta)] \end{equation} $

I will not cover what all of these parameters are, but you can certainly research this.

Observe the following example of how a Bernoulli distribution is in the GLM family:

$ P(y=1) = \phi\\ P(y=0) = 1 - \phi\\ P(y) = \phi^y(1-\phi)^{1-y} = \text{exp}(y\text{log}(\phi) + (1-y)\text{log}(1-\phi))\\ = \text{exp}(y\text{log}(\phi) + \text{log}(1-\phi)-y\text{log}(1-\phi))\\ = \text{exp}(y\text{log}(\frac{\phi}{1-\phi}) + \text{log}(1-\phi)) $

You can see that in this case,

$ b(y) = 1\\ T(y) = y\\ \eta = \text{log}(\frac{\phi}{1-\phi})\\ a(\eta) = -\text{log}(1-\phi) $

Notice what happens when we solve for $\phi$ in terms of $\eta$:

$ \eta = \text{log}(\frac{\phi}{1-\phi})\\ e^\eta =\frac{\phi}{1-\phi}\\ e^{-\eta} = \frac{1-\phi}{\phi} = \frac{1}{\phi}-1\\ e^{-\eta}+1 = \frac{1}{\phi}\\ \phi = [\text{exp}(-{\eta})+1]^{-1} $

So to get $\phi=P(y=1)$, we take the sigmoid of $\eta$. The design choice comes in to play when we assume that $\eta = w^Tx$, where $w$ are your weights and $x$ is your data, both of which we assume to be $\in\mathbb{R}^n$. By making this assumption, we can fit $w$ to approximate $\phi$.

If you were to go through this same process for a Multinoulli distribution, you would end up deriving the softmax function.