What is an intuitive interpretation for the softmax transformation?

A recent question on this site asked about the intuition of softmax regression. This has inspired me to ask a corresponding question about the intuitive meaning of the softmax transformation itself. The general scaled form of the softmax function $$\mathbf{S}: \bar{\mathbb{R}}^{n-1} \times \mathbb{R}_+ \rightarrow \Delta^n$$ is given by:

$$\mathbf{S}(\mathbf{z}, \lambda) \equiv \Bigg( \frac{1}{1 + \sum_{k=1}^{n-1} \exp(\lambda z_k)}, \frac{\exp(\lambda z_1)}{1 + \sum_{k=1}^{n-1} \exp(\lambda z_k)}, \ \cdots \ , \frac{\exp(\lambda z_{n-1})}{1 + \sum_{k=1}^{n-1} \exp(\lambda z_k)} \Bigg).$$

Is there any intuitive interpretation that describes the mapping from the extended-real value $$\mathbf{z} \in \bar{\mathbb{R}}^{n-1}$$ to the corresponding probability vector $$\mathbf{p} \in \Delta^n$$ in simple terms? I.e., is there some explanation that can describe the mapping in an intuitive way by use of geometry, analogy, etc.?

Intuition is a funky concept. For an ex-physicist, myself, seeing softmwax for the first time was "Ok, this is Boltzmann distribution." For a statistician it would be "Oh, isn't this mlogit?"

Physicist's intuition

Softmax is literally the case of canonical ensemble: $$p_i=\frac 1 Q e^{- {\varepsilon}_i / (k T)}=\frac{e^{- {\varepsilon}_i / (kT)}}{\sum_{j=1}^{n}{e^{- {\varepsilon}_j / (k T)}}}$$ The denominator is called a canonical partition function, it's basically a normalizing constant to make sure the probabilities add up to 100%. But it has a physical meaning too: the system can only be in one of its M states, that's why probabilities must add up. This stuff is straight up from statistical mechanics.

The probability of a state $$i$$ is defined by its energy $$\varepsilon_i$$ relative to the energies of all other states. You see, in physics systems always try to minimize the energy, so the probability of the state with the lowest energy must be the highest. However, if the temperature of the system $$T$$ is high, then the difference in probabilities of the lowest energy state and other states will vanish: $$\lim_{T\to\infty}p_{min}/p_i=\lim_{T\to\infty}e^{ ({\varepsilon_i- \varepsilon}_{min}) / (k T)}=1$$

So, in OP's equation the energy $$\varepsilon=-z$$ and the temperature is $$T\sim 1/\lambda$$. He also isolates the base state, and sets its probability with 1 instead of the exponential. This doesn't change anything for intuition, it only sets all energies relative to a chosen base state. This is VERY intuitive to a physicist.

Statistician's intuition

A statistician will immediately recognize the multinomial logit regression. For those who only know bivariate logit regression, here's how mlogit works.

Estimate $$n-1$$ bivariate logits of $$n-1$$ states vs a chosen base state on the censored data set. So, you create a dataset from a base state, say 1, and one of the states $$i\in[2,n]$$. This way you get $$n-1$$ logits for each $$i$$, conditional ones: $$\ln\frac{Pr[i|i\cup 1]}{Pr[1|i\cup 1]}\sim X_i$$ This equation is more recognizable as: $$\ln\frac{p}{1-p}\sim X_i$$ This is how it is usually presented in bivariate cases, where there are only categories to choose from, like in our censored subset of the full dataset with $$n$$ categories.

Using Bayes theorem we know that: $$Pr[i|i\cup 1]=\frac{Pr[i]}{Pr[i]+Pr}$$ So, we can trivially combine $$n-1$$ bivariate regressions into a single one to get unconditional probabilities: $$Pr[i]=\frac{e^{X_i\beta_i}}{1+\sum_i e^{X_i\beta_i}}$$ This gets us OP's equation.

• Great answer, but your link to the multinomial logit regression seems to be broken. Nov 17 '21 at 14:39
• @mhovd pointed the link to R package, thanks for flagging it Nov 17 '21 at 15:17

The presence of the exponentials in this function makes it fairly easy to construct an intuitive meaning for the transformation in terms of exponential growth of a set of quantities. Consequently, I will give an intuitive description for the function in terms of a simple financial portfolio earning returns over time. This can be modified or generalised to refer to any similar example involving a set of quantities affected by exponential growth.

A simple intuitive interpretation: Suppose you have an initial investment portfolio consisting of a set of $$n$$ investment items with the same value. (Without loss of generality, set the initial values of each item to one.) The first item always earns zero return, and the remaining items earn fixed continuous rates-of-return of $$z_1, ..., z_{n-1}$$ in each time period (these returns may be positive or negative or zero). Now, after $$\lambda$$ time units the first item will have a value of one, and the remaining items will have respective values $$\exp(\lambda z_1), ..., \exp(\lambda z_{n-1})$$. The total value of the portfolio is $$1 + \sum_{k=1}^{n-1} \exp(\lambda z_k)$$. Consequently, the softmax function gives you the vector of proportions of the size of each item in the portfolio after $$\lambda$$ time units have elapsed.

$$\mathbf{S}(\mathbf{z}) = \text{Proportion vector for size of items in portfolio after } \lambda \text{ time units}.$$

This gives a simple intuitive interpretation of the softmax transformation. It is also worth noting that one can easily construct a corresponding intuitive interpretation for the inverse-softmax transformation. The latter transformation would take in the proportion vector showing the relative sizes of the items in the portfolio, and it would figure out the continuous rates-of-return that led to that outcome over $$\lambda$$ time units.

This is just one intuitive interpretation for the softmax function, using a finance context. One can easily construct corresponding interpretations for any finite set of initial items that are each subject to exponential growth over time (with one item fixed to have zero growth).

At least for deep learning purposes, one shouldn't overthink the importance of the exact terms in the softmax function, nor that it maps onto the probability simplex. What matters is that it's a function $$s_\lambda : \mathbb{R}^n \to [0,1]^n$$ with the properties \begin{align} \frac{\partial (s_\lambda(\mathbf{z}))_i}{\partial z_i} >&\: 0 & \text{for all i} \\ \frac{\partial (s_\lambda(\mathbf{z}))_i}{\partial z_j} <&\: 0 & \text{for all j\neq i} \end{align} and $$s_\lambda(\mathbf{z}) \approx \{0,0,...,\underbrace{1}_{i},0,...\}$$ if $$z_i \gg z_j$$ for all $$j\neq i$$. As a result, tweaking the inputs sufficiently far in some direction will eventually give you a nearly categorical output, and the gradients of mismatching results are always propagated in a direction that's suitable.

So why is this exponential form the softmax function that's used by everyone? It's mostly just that $$\exp$$ happens to be both monotone and strictly convex, and from that the above properties already follow. In addition, it's very fast-growing, which means the convergence to almost-black&white result does not take too long time.

One way to think about the softmax function is that it gives you an output that can be interpreted as a probability distribution (i.e., all numbers are in the range [0,1], and they sum to 1). This is useful, because then the output of the softmax can be interpreted as a "probability" of each class/category (conditioned on the features).

Why does its output always have this property? Well, the softmax is essentially the composition of two steps:

1. Apply the exp function to each value. This makes all values non-negative.
2. Normalize the values so they sum to 1 (by dividing by the sum). This makes all values sum to 1.

After both of these steps, you are guaranteed that all values are non-negative and they sum to 1, which means they can be interpreted as a probability distribution.

The generalized softmax with scaling $$\lambda$$ just amounts to multiplying all values by $$\lambda$$, then applying the softmax, so it is not very different from the normal softmax.

Another way to think about the softmax is that it is a natural generalization of the standard logistic function $$f(x) = e^x/(1+e^x)$$, used in logistic regression. Logistic regression is used when you want to do two-class classification. When you want to do multi-class classification, you replace the standard logistic function with the softmax function. If you apply the softmax function with two classes, the result reduces to the standard logistic function that you're used to in (two-class) logistic regression.