Logistic regression for multiclass

Question

I got the model for the logistic regression for multiclass which is given by

$$ P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^TX^{(i)})}{1+ \sum_{m=1}^{k}\exp(\theta_m^T X^{(i)})} $$

where k is the number of classes theta is the parameter to be estimated j is the jth class Xi is the training data

Well one thing I didn't get is how come the denominator part $$ 1+ \sum_{m=1}^{k}\exp(\theta_m^T X^{(i)}) $$ normalized the model. I mean it makes the probability stay between 0 and 1.

I mean I am used to logistic regression being

$$ P(Y=1|X^{(i)}) = 1/ (1 + \exp(-\theta^T X^{(i)})) $$

Actually, I am confused with the nomalization thing. In this case since it is a sigmoid function it never lets the value be less than 0 or greater than 1. But I am confused in the multi class case. Why is it so?

This is my reference https://list.scms.waikato.ac.nz/pipermail/wekalist/2005-February/029738.html. I think it should have been to be normalizing $$ P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^T X^{(i)})}{\sum_{m=1}^{k} \exp(\theta_m^T X^{(i)})} $$

Answer 1

Your formula is wrong (the upper limit of the sum). In logistic regression with $K$ classes ($K> 2$) you basically create $K-1$ binary logistic regression models where you choose one class as reference or pivot. Usually, the last class $K$ is selected as the reference. Thus, the probability of the reference class can be calculated by $$P(y_i = K | x_i) = 1 - \sum_{k=1}^{K-1} P(y_i = k | x_i) .$$ The general form of the probability is $$P(y_i = k | x_i) = \frac{\exp(\theta_i^T x_i)}{\sum_{i=1}^K \exp(\theta_i^T x_i)} .$$ As the $K$-th class is your reference $\theta_K = (0, \ldots, 0)^T$ and therefore $$\sum_{i=1}^K \exp(\theta_i^T x_i) = \exp(0) + \sum_{i=1}^{K-1} \exp(\theta_i^T x_i) = 1 + \sum_{i=1}^{K-1} \exp(\theta_i^T x_i) .$$ In the end you get the following formula for all $k < K$: $$ P(y_i = k | x_i) = \frac{\exp(\theta_i^T x_i)}{1 + \sum_{i=1}^{K-1} \exp(\theta_i^T x_i)} $$

Answer 2

I think you're being confused by a typo: Your $k$ should be $k-1$ in the first equation. The 1's you see in the logistic case are actually $\exp(0)$s, e.g., when there is a $k$th $\theta=0$.

Assume that $\theta_1 X=b$. Now notice that you can get from the last formulation to the logistic regression version like $$ \frac{\exp(b)}{\exp(0)+\exp(b)} = \frac{\exp(0)}{\exp(0)+\exp(-b)} = \frac{1}{1+\exp(-b)} $$ For multiple classes, just replace the denominator in the first two quantities by a sum over exponentiated linear predictors.