4
$\begingroup$

Suppose we have data matrix $X$ which is $n\times p$ matrix ($n$ data points, and $p$ features including intercept), i.e., we already add the intercept term (append $1$ column) to $X$

  • For logistic regression (binary classification), the model parameters / regression coefficients $\beta$ is a length $p$ vector.

  • For softmax regression (multi-class classification), the model parameters $W$ is $p \times m$ matrix, where $m$ is the number of classes.

Now, suppose we set $m=2$, then $W$ is a $p \times 2$ matrix. They are the same model, so the number of parameters CANNOT be doubled and we should be able to derive one from another.

So, my question is

What is the relationship between $\beta$ and $W$?

$\endgroup$
5
$\begingroup$

Suppose you have a binary classification problem with $p$ features (including bias) and you do Multi-class regression with softmax activation. Then, the probability of an observation, $x,$ representing class $1$ is,

$$ \begin{split} p_1(x) &= \frac{\exp(\beta_1^T x)}{\exp(\beta_1^T x) + \exp(\beta_2^T x)} \\ &= \frac{1}{1 + \exp[(\beta_2 - \beta_1)^T x]} = \sigma\left([\beta_1 - \beta_2]^T x\right). \end{split} $$ where both $\beta_1$ and $\beta_2$ are $p \times 1$ parameters, and I simply multiplied both numerator and denominator by $\exp(- \beta_1^T x).$ Note that the RHS of the above expression is simply the sigmoid function. Thus, the $p \times 1$ parameter $\beta$ you would get from doing ordinary binary logistic regression with sigmoid activation is analogous to $\beta_1 - \beta_2,$ where the latter parameters are from multiclass softmax regression.

$\endgroup$
6
$\begingroup$

In multinomial regression we model odds of observing $Y=k$ for each of the $K-1$ classes relatively to the $K$-th class. So with $K=2$ the model reduces to logistic regression.

Let me quote Wikipedia:

One fairly simple way to arrive at the multinomial logit model is to imagine, for $K$ possible outcomes, running $K-1$ independent binary logistic regression models, in which one outcome is chosen as a "pivot" and then the other $K-1$ outcomes are separately regressed against the pivot outcome. This would proceed as follows, if outcome $K$ (the last outcome) is chosen as the pivot:

$$ \begin{align} \ln \frac{\Pr(Y_i=1)}{\Pr(Y_i=K)} &= \boldsymbol\beta_1 \cdot \mathbf{X}_i \\ \ln \frac{\Pr(Y_i=2)}{\Pr(Y_i=K)} &= \boldsymbol\beta_2 \cdot \mathbf{X}_i \\ \cdots & \cdots \\ \ln \frac{\Pr(Y_i=K-1)}{\Pr(Y_i=K)} &= \boldsymbol\beta_{K-1} \cdot \mathbf{X}_i \\ \end{align} $$

(...) Using the fact that all $K$ of the probabilities must sum to one, we find:

$$ \Pr(Y_i=K) = 1 - \sum_{k=1}^{K-1}{\Pr(Y_i=K)}e^{\boldsymbol\beta_k \cdot \mathbf{X}_i} \Rightarrow \Pr(Y_i=K) = \frac{1}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} $$

We can use this to find the other probabilities:

$$ \begin{align} \Pr(Y_i=1) &= \frac{e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \Pr(Y_i=2) &= \frac{e^{\boldsymbol\beta_2 \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \cdots & \cdots \\ \Pr(Y_i=K-1) &= \frac{e^{\boldsymbol\beta_{K-1} \cdot \mathbf{X}_i}}{1 + \sum_{k=1}^{K-1} e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}} \\ \end{align} $$

With logistic regression you have only

$$ \ln \frac{\Pr(Y_i=1)}{\Pr(Y_i=0)} = \boldsymbol\beta \cdot \mathbf{X}_i $$

and probabilities derived from it. Logistic regression and multinomial regression with $K=2$ are the same thing.

Example

If you take the data from this online tutorial and run logistic regression and multinomial regression on it, you'll see exactly the same results (compare the coefficients):

library(nnet)
mydata <- read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
mydata$admit2 <- ifelse(mydata$admit == 1, 0, 1)
## 
## glm(admit ~ gre + gpa, data = mydata, family = "binomial")
## 
## Call:  glm(formula = admit ~ gre + gpa, family = "binomial", data = mydata)
## 
## Coefficients:
## (Intercept)          gre          gpa  
##   -4.949378     0.002691     0.754687  
## 
## Degrees of Freedom: 399 Total (i.e. Null);  397 Residual
## Null Deviance:       500 
## Residual Deviance: 480.3     AIC: 486.3

multinom(cbind(admit2, admit) ~ gre + gpa, data = mydata, family = "binomial")
## 
## # weights:  8 (3 variable)
## initial  value 277.258872 
## final  value 240.171991 
## converged
## Call:
## multinom(formula = cbind(admit2, admit) ~ gre + gpa, data = mydata, 
##     family = "binomial")
## 
## Coefficients:
##       (Intercept)         gre       gpa
## admit   -4.949375 0.002690691 0.7546848
## 
## Residual Deviance: 480.344 
## AIC: 486.344 
$\endgroup$
  • 1
    $\begingroup$ thanks for the answer, but i think it did not answer my question. I know they are the same, and trying to ask how to derive $\beta$ from $W$. $\endgroup$ – hxd1011 Aug 22 '17 at 13:28
  • 2
    $\begingroup$ @hxd1011 but they are the same thing, so you don't have to derive anything... $\endgroup$ – Tim Aug 22 '17 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.