1
$\begingroup$

Given the standard multinomial logistic regression model with reference class K:

$P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^TX^{(i)})}{1+ \sum_{m=1}^{K-1}\exp(\theta_m^T X^{(i)})}$

Would I run into any problems if I estimate K-1 binary regressions independently (R: glm(,family = "binomial") vs. nnet::multinom()) on filtered data sets where the outcomes are K and j? This implies there will be different sets of predictors and possibly different transformations (e.g. floors, caps, splines, etc.).

$\endgroup$
3
$\begingroup$

You will get close, except that multinomial logit will enforce the constraint that all predicted probabilities add up to one, while different logistic regressions cannot enforce that constraint.

$\endgroup$
  • $\begingroup$ With $P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^TX^{(i)})}{1+ \sum_{m=1}^{K-1}\exp(\theta_m^T X^{(i)})}$ and $P(Y=K|X^{(i)}) = \frac{1}{1+ \sum_{m=1}^{K-1}\exp(\theta_m^T X^{(i)})}$, all probabilities do add to 1. No? $\endgroup$ – Bebo Jan 6 '17 at 18:39
  • $\begingroup$ If you estimate your model with a series of separate binary logistic regressions, then there is no way to impose that constraint. Only when you estimate all probabilities in one model can you impose that constraint, which is what multinomial logistic regression does. $\endgroup$ – Maarten Buis Jan 6 '17 at 20:07
  • $\begingroup$ Thank you for responding! Do you think that I have an error in my response to the comment below (Taylor)? $\endgroup$ – Bebo Jan 6 '17 at 20:14
  • $\begingroup$ Maybe the constraint could be imposed by some metod of constrained optimization? $\endgroup$ – kjetil b halvorsen Apr 3 '18 at 11:15
  • $\begingroup$ Maybe, but why bother if you can just do this with regular ML in any standard statistics program? $\endgroup$ – Maarten Buis Apr 3 '18 at 18:57
1
$\begingroup$

I think you made a small typo. The sum in the denominator should be running from $1$ to $k-1$, not $k$. Edit: nevermind you fixed it :)

Fitting $k-1$ independent regressions, you get estimates for $\{\theta_j\}$ with $j=1, \ldots, k-1$. The $k-1$ models are parametrized as: $$ P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^TX^{(i)})}{1+ \exp(\theta_j^T X^{(i)})}. $$ Looking at each row, the dependent variable will either be a succes ($=j$) or not $(\neq j)$. You're basically ignoring all the other outcomes here.

On the other hand if you fit a multinomial logistic regression, it will enforce, as @MaartenBuis says, the constraint that all the probabilities sum to $1$. You're also assuming, probably more correctly, that each dependent variable can be any value from $1$ to $k$, and it will follow a multinomial distribution with the parameter vector being dependent on the $X^{(i)}$ data. In other words, the model assumed is $$ P(Y=j|X^{(i)}) = \frac{\exp(\theta_j^TX^{(i)})}{1+ \sum_{m=1}^{k-1}\exp(\theta_m^T X^{(i)})} $$ with $j=1,\ldots,K-1$, and you still get the same amount of coefficient estimates. But you can see that $$ \sum_{i=1}^k P(Y=i|X^{(i)}) = \sum_{i=1}^{k-1} \frac{\exp(\theta_j^TX^{(i)})}{1+ \sum_{m=1}^{k-1}\exp(\theta_m^T X^{(i)})} + \frac{1}{1+\sum_{m=1}^{k}\exp(\theta_m^T X^{(i)})} = 1. $$

I can't prove when the estimates between these two procedures will be close or not, but I can tell you that these two procedures are assuming totally different models to be true.

$\endgroup$
  • $\begingroup$ The key is that we filter all observations with outcomes other than K and j. Thus we have: $ln(\frac{P(y=j)}{P(y=K)})=ln(\frac{P(y=j)}{1-\sum_{i=1}^{K-1}P(y=i)})=ln(\frac{P(y=j)}{1-P(y=j)-\sum_{i=1;i \neq j}^{K-1}P(y=i)})$. However, the last sum in the denominator is zero due to the filtering of the data and becomes a binary regression. Thus for a 5-category case we would do: reg_1 = glm(my_frml, data=dt[y %in% c(1,5), ], family='binomial'). Am I missing anything? $\endgroup$ – Bebo Jan 6 '17 at 19:17
  • $\begingroup$ Filtering? What's zero? $\endgroup$ – Taylor Jan 9 '17 at 20:06
1
$\begingroup$

Just for the record - apparently my original question referred to a methodology first described in an article by Begg and Gray "Calculation of polychotomous logistic regression parameters using individualized regressions" from 1984. Hosmer mentions it on page 277 of Applied Logistic Regression...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.