Definition of softmax function

This question follows up on stats.stackexchange.com/q/233658

The logistic regression model for classes {0, 1} is

$$\mathbb{P} (y = 1 \;|\; x) = \frac{\exp(w^T x)}{1 + \exp(w^T x)} \\ \mathbb{P} (y = 0 \;|\; x) = \frac{1}{1 + \exp(w^T x)}$$

Clearly those probabilities sum to 1. By setting $w = \beta_1 - \beta_0$ we could also define logistic regression as

$$\mathbb{P} (y = c \;|\; x) = \frac{\exp(\beta_c^T x)}{\exp(\beta_0^T x) + \exp(\beta_1^T x)} \quad \forall \; c \in \{0, 1\}$$

However, the second definition is rarely used because the coefficients $\beta_0$ and $\beta_1$ are not unique. In other words, the model is not identifiable, just like linear regression with two variables that are multiples of each other.

Question

In machine learning, why is the softmax regression model for classes {0, 1, ..., K – 1} usually defined as follows?

$$\mathbb{P} (y = c \;|\; x) = \frac{\exp(\beta_c^T x)}{\exp(\beta_0^T x) + \dots + \exp(\beta_{K-1}^T x)} \quad \forall \; c \in \{0, \dots, K-1\}$$

Shouldn't it instead be

\begin{align*} \mathbb{P} (y = c \;|\; x) &= \frac{\exp(w_c^T x)}{1 + \exp(w_1^T x) + \dots + \exp(w_{K-1}^T x)} \quad \forall \; c \in \{1, \dots, K-1\} \\ \mathbb{P} (y = 0 \;|\; x) &= \frac{1}{1 + \exp(w_1^T x) + \dots + \exp(w_{K-1}^T x)} \end{align*}

Side note: In statistics, softmax regression is called multinomial logistic regression and the classes are {1, ..., K}. I find this a bit awkward because when K = 2, the classes are {1, 2} instead of {0, 1} so it's not exactly a generalization of logistic regression.

• Is it usually defined that way? Can you point to a reference? – The Laconic Jun 2 '18 at 2:37
• @TheLaconic See the definition of softmax at scikit-learn.org/stable/modules/neural_networks_supervised.html and www.tensorflow.org/versions/r1.1/get_started/mnist/beginners – farmer Jun 2 '18 at 6:46
• OK. I asked because I've never seen MNL regression models defined in this way. But apparently it's "usual" in the context of neural nets--and now I have the same question as you. – The Laconic Jun 2 '18 at 13:12
• I'm not sure how to answer a "shouldn't it be X" question. The given definition is differentiable, and defines a probability distribution (sums to 1). It seems like those are the important parts, so why shouldn't it be that instead of the other? – kbrose Sep 18 '18 at 14:38
• @kbrose because of the lack of identiability – Taylor Jun 26 at 23:56

Yes, you are correct that there is a lack of identifiability unless one of the coefficent vectors is fixed. There are some reasons that don't mention this. I can't speak to why they omit this detail, but here's an explanation of what it is and how to fix it.

Description

Say you have observations $$y_i \in \{0, 1, 2, \ldots, K-1\}$$ and predictors $$\mathbf{x}_i^\intercal \in \mathbb{R}^p$$, where $$i$$ goes from $$1$$ to $$n$$ and denotes the observation number/index. You will need to estimate $$K$$ $$p$$-dimensional coefficient vectors $$\boldsymbol{\beta}^0, \boldsymbol{\beta}^1, \ldots, \boldsymbol{\beta}^{K-1}$$.

The softmax function is indeed defined as $$\text{softmax}(\mathbf{z})_i = \frac{\exp(z_i)}{\sum_{l=0}^{K-1}\exp(z_l)},$$ which has nice properties such as differentiability, it sums to $$1$$, etc.

Multinomial logistic regression uses the softmax function for each observation $$i$$ on the vector $$\begin{bmatrix} \mathbf{x}_i^\intercal \boldsymbol{\beta}^0 \\ \mathbf{x}_i^\intercal \boldsymbol{\beta}^1 \\ \vdots \\ \mathbf{x}_i^\intercal \boldsymbol{\beta}^{K-1}, \end{bmatrix}$$

which means $$\begin{bmatrix} P(y_i = 0) \\ P(y_i = 1) \\ \vdots \\ P(y_i = K-1) \end{bmatrix} = \begin{bmatrix} \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^0] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \\ \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^1] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \\ \vdots \\ \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^{K-1}] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \end{bmatrix}.$$

The problem

However, the likelihood is not identifiable because multiple parameter collections will give the same likelihood. For example, shifting all the coefficient vectors by the same vector $$\mathbf{c}$$ will produce the same likelihood. This can be seen if you multiply each the numerator and denominator of each element of the vector by a constant $$\exp[-\mathbf{x}_i^\intercal \mathbf{c}]$$, nothing changes:

$$\begin{bmatrix} \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^0] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \\ \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^1] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \\ \vdots \\ \frac{\exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^{K-1}] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal \boldsymbol{\beta}^k] } \end{bmatrix} = \begin{bmatrix} \frac{\exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^0-\mathbf{c})] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^k-\mathbf{c})] } \\ \frac{\exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^1-\mathbf{c})] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^k-\mathbf{c})] } \\ \vdots \\ \frac{\exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^{K-1} - \mathbf{c})] }{ \sum_{k=0}^{K-1} \exp[\mathbf{x}_i^\intercal (\boldsymbol{\beta}^k -\mathbf{c}) ] } \end{bmatrix}.$$

Fixing it

The way to fix this is to constrain the parameters. Fixing one of them will lead to identifiability, because shifting all of them will no longer be permitted.

There are two common choices:

• set $$\mathbf{c} = \boldsymbol{\beta}^0$$, which means $$\boldsymbol{\beta}^0 = \mathbf{0}$$ (you mention this one), and
• set $$\mathbf{c} = \boldsymbol{\beta}^{K-1}$$, which means $$\boldsymbol{\beta}^{K-1} = \mathbf{0}$$.

Ignoring it

Sometimes the restriction isn't necessary, though. For instance, if you were interested in forming a confidence interval for the quantity $$\beta^0_1 - \beta^2_1$$, then this is the same as $$\beta^0_1 - c - [\beta^2_1-c]$$, so inference on relatively quantities doesn't really matter. Also, if your task is prediction instead of parameter inference, your predictions will be unaffected if all coefficient vectors are estimated (without constraining one).