# Relationship between logistic regression and Softmax Regression with 2 classes

Suppose we have data matrix $X$ which is $n\times p$ matrix ($n$ data points, and $p$ features including intercept), i.e., we already add the intercept term (append $1$ column) to $X$

• For logistic regression (binary classification), the model parameters / regression coefficients $\beta$ is a length $p$ vector.

• For softmax regression (multi-class classification), the model parameters $W$ is $p \times m$ matrix, where $m$ is the number of classes.

Now, suppose we set $m=2$, then $W$ is a $p \times 2$ matrix. They are the same model, so the number of parameters CANNOT be doubled and we should be able to derive one from another.

So, my question is

What is the relationship between $\beta$ and $W$?

Suppose you have a binary classification problem with $p$ features (including bias) and you do Multi-class regression with softmax activation. Then, the probability of an observation, $x,$ representing class $1$ is,

$$\begin{split} p_1(x) &= \frac{\exp(\beta_1^T x)}{\exp(\beta_1^T x) + \exp(\beta_2^T x)} \\ &= \frac{1}{1 + \exp[(\beta_2 - \beta_1)^T x]} = \sigma\left([\beta_1 - \beta_2]^T x\right). \end{split}$$ where both $\beta_1$ and $\beta_2$ are $p \times 1$ parameters, and I simply multiplied both numerator and denominator by $\exp(- \beta_1^T x).$ Note that the RHS of the above expression is simply the sigmoid function. Thus, the $p \times 1$ parameter $\beta$ you would get from doing ordinary binary logistic regression with sigmoid activation is analogous to $\beta_1 - \beta_2,$ where the latter parameters are from multiclass softmax regression.

In multinomial regression we model odds of observing $Y=k$ for each of the $K-1$ classes relatively to the $K$-th class. So with $K=2$ the model reduces to logistic regression.

Let me quote Wikipedia:

One fairly simple way to arrive at the multinomial logit model is to imagine, for $K$ possible outcomes, running $K-1$ independent binary logistic regression models, in which one outcome is chosen as a "pivot" and then the other $K-1$ outcomes are separately regressed against the pivot outcome. This would proceed as follows, if outcome $K$ (the last outcome) is chosen as the pivot:

\begin{align} \ln \frac{\Pr(Y_i=1)}{\Pr(Y_i=K)} &= \boldsymbol\beta_1 \cdot \mathbf{X}_i \\ \ln \frac{\Pr(Y_i=2)}{\Pr(Y_i=K)} &= \boldsymbol\beta_2 \cdot \mathbf{X}_i \\ \cdots & \cdots \\ \ln \frac{\Pr(Y_i=K-1)}{\Pr(Y_i=K)} &= \boldsymbol\beta_{K-1} \cdot \mathbf{X}_i \\ \end{align}

(...) Using the fact that all $K$ of the probabilities must sum to one, we find:

$$\Pr(Y_i=K) = 1 - \sum_{k=1}^{K-1}{\Pr(Y_i=K)}e^{\boldsymbol\beta_k \cdot \mathbf{X}_i} \Rightarrow \Pr(Y_i=K) = \frac{1}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}}$$

We can use this to find the other probabilities:

\begin{align} \Pr(Y_i=1) &= \frac{e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \Pr(Y_i=2) &= \frac{e^{\boldsymbol\beta_2 \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \cdots & \cdots \\ \Pr(Y_i=K-1) &= \frac{e^{\boldsymbol\beta_{K-1} \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \end{align}

With logistic regression you have only

$$\ln \frac{\Pr(Y_i=1)}{\Pr(Y_i=0)} = \boldsymbol\beta \cdot \mathbf{X}_i$$

and probabilities derived from it. Logistic regression and multinomial regression with $K=2$ are the same thing.

## Example

If you take the data from this online tutorial and run logistic regression and multinomial regression on it, you'll see exactly the same results (compare the coefficients):

library(nnet)
mydata$admit2 <- ifelse(mydata$admit == 1, 0, 1)
##
## glm(admit ~ gre + gpa, data = mydata, family = "binomial")
##
## Call:  glm(formula = admit ~ gre + gpa, family = "binomial", data = mydata)
##
## Coefficients:
## (Intercept)          gre          gpa
##   -4.949378     0.002691     0.754687
##
## Degrees of Freedom: 399 Total (i.e. Null);  397 Residual
## Null Deviance:       500
## Residual Deviance: 480.3     AIC: 486.3

multinom(cbind(admit2, admit) ~ gre + gpa, data = mydata, family = "binomial")
##
## # weights:  8 (3 variable)
## initial  value 277.258872
## final  value 240.171991
## converged
## Call:
## multinom(formula = cbind(admit2, admit) ~ gre + gpa, data = mydata,
##     family = "binomial")
##
## Coefficients:
##       (Intercept)         gre       gpa

• thanks for the answer, but i think it did not answer my question. I know they are the same, and trying to ask how to derive $\beta$ from $W$. – Haitao Du Aug 22 '17 at 13:28