34
$\begingroup$

Lets say we have a dependent variable $Y$ with few categories and set of independent variables.

What are the advantages of multinomial logistic regression over set of binary logistic regressions (i.e. one-vs-rest scheme)? By set of binary logistic regression I mean that for each category $y_{i} \in Y$ we build separate binary logistic regression model with target=1 when $Y=y_{i}$ and 0 otherwise.

$\endgroup$
  • 3
    $\begingroup$ Mathematically, a multinomial logit model is a set of binary logit models, all compared against a base alternative. But because you get to collapse generic parameters and maybe combine some others, the MNL will always be at least as efficient (and probably more so). I see no reason to ever use a series of binomial models. $\endgroup$ – gregmacfarlane Mar 14 '13 at 11:52
  • 2
    $\begingroup$ @gmacfarlane: I've tried to simulate data where MNL would be better than series of binary logistic regressions, but every time on average the quality was the same. I was comparing lift charts and after averaging results from few simulations they looke almost the same. Maybe You have an idea how to generate data so MNL beats binary logistic regressions? Although MNL had a great advantage, its scores could be interpreted as probability. $\endgroup$ – Tomek Tarczynski Mar 14 '13 at 14:16
  • $\begingroup$ Multinomial Logistic regression is the extension of binary logit regression. It is used when the dependent variables of the study is three and above, whereas, binary logit is used when the dependent variables of the study is two. $\endgroup$ – user48573 Jun 18 '14 at 12:20
  • $\begingroup$ To reader: I recommend starting at @julieth's answer and following up by reading ttnphns'. I think the former more directly answers the original question but the latter adds some interesting context. ttnphns also shows the different features that are available for both in a popular software routine, which could itself constitute a reason to use one over the other (see gregmacfarlane's statement). $\endgroup$ – Ben Ogorek Feb 16 at 16:42
20
$\begingroup$

If $Y$ has more than two categories your question about "advantage" of one regression over the other is probably meaningless if you aim to compare the models' parameters, because the models will be fundamentally different:

$\bf log \frac{P(i)}{P(not~i)}=logit_i=linear~combination$ for each $i$ binary logistic regression, and

$\bf log \frac{P(i)}{P(r)}=logit_i=linear~combination$ for each $i$ category in multiple logistic regression, $r$ being the chosen reference category ($i \ne r$).

However, if your aim is only to predict probability of each category $i$ either approach is justified, albeit they may give different probability estimates. The formula to estimate a probability is generic:

$\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+exp(logit_j)+\dots+exp(logit_r)}$, where $i,j,\dots,r$ are all the categories, and if $r$ was chosen to be the reference one its $\bf exp(logit)=1$. So, for binary logistic that same formula becomes $\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+1}$. Multinomial logistic relies on the (not always realistic) assumption of independence of irrelevant alternatives whereas a series of binary logistic predictions does not.


A separate theme is what are technical differences between multinomial and binary logistic regressions in case when $Y$ is dichotomous. Will there be any difference in results? Most of the time in the absence of covariates the results will be the same, still, there are differences in the algorithms and in output options. Let me just quote SPSS Help about that issue in SPSS:

Binary logistic regression models can be fitted using either the Logistic Regression procedure or the Multinomial Logistic Regression procedure. Each procedure has options not available in the other. An important theoretical distinction is that the Logistic Regression procedure produces all predictions, residuals, influence statistics, and goodness-of-fit tests using data at the individual case level, regardless of how the data are entered and whether or not the number of covariate patterns is smaller than the total number of cases, while the Multinomial Logistic Regression procedure internally aggregates cases to form subpopulations with identical covariate patterns for the predictors, producing predictions, residuals, and goodness-of-fit tests based on these subpopulations. If all predictors are categorical or any continuous predictors take on only a limited number of values—so that there are several cases at each distinct covariate pattern—the subpopulation approach can produce valid goodness-of-fit tests and informative residuals, while the individual case level approach cannot.

Logistic Regression provides the following unique features:

• Hosmer-Lemeshow test of goodness of fit for the model

• Stepwise analyses

• Contrasts to define model parameterization

• Alternative cut points for classification

• Classification plots

• Model fitted on one set of cases to a held-out set of cases

• Saves predictions, residuals, and influence statistics

Multinomial Logistic Regression provides the following unique features:

• Pearson and deviance chi-square tests for goodness of fit of the model

• Specification of subpopulations for grouping of data for goodness-of-fit tests

• Listing of counts, predicted counts, and residuals by subpopulations

• Correction of variance estimates for over-dispersion

• Covariance matrix of the parameter estimates

• Tests of linear combinations of parameters

• Explicit specification of nested models

• Fit 1-1 matched conditional logistic regression models using differenced variables

$\endgroup$
  • $\begingroup$ I know that these models will be different, but I don't know which one is better in which situation. I will ask the question in another way. If You were given a task: For each person predict the probability that some mobile phone company is the favourite one (lets assume every one have favourite mobile phone company). Which of those methods would You use and what are the advantages over the second one? $\endgroup$ – Tomek Tarczynski Mar 14 '13 at 9:40
  • $\begingroup$ @Tomek I expanded my answer a little bit $\endgroup$ – ttnphns Mar 14 '13 at 17:31
  • $\begingroup$ Though I think @julieth's is the best answer to O.P.'s original question, I do owe you for the introduction to the Independence of Irrelevant Alternatives assumption. One question I still have is whether separate logistics truly get around it; the Wikipedia article you linked to mentioned probit and "nested logit" as allowing violations of IIA $\endgroup$ – Ben Ogorek Feb 16 at 16:36
  • $\begingroup$ Would you be able to explain how to fit the models with a choice of reference category? For category $i$, do we only use a subset of the data that is either in the reference category $r$ or the category $i$, for $i \neq r$? $\endgroup$ – user21359 May 6 at 21:19
13
$\begingroup$

Because of the title, I'm assuming that "advantages of multiple logistic regression" means "multinomial regression". There are often advantages when the model is fit simultaneously. This particular situation is described in Agresti (Categorical Data Analysis, 2002) pg 273. In sum (paraphrasing Agresti), you expect the estimates from a joint model to be different than a stratified model. The separate logistic models tend to have larger standard errors although it may not be so bad when the most frequent level of the outcome is set as the reference level.

$\endgroup$
  • $\begingroup$ Thanks! I will try to find this book, unfortunatelly google.books provides content only till page 268. $\endgroup$ – Tomek Tarczynski Mar 14 '13 at 13:39
  • $\begingroup$ @TomekTarczynski I summarized the relevant information from the paragraph, so you may not get any more info related to this question from looking at the book (although the book is great so you will get other good info). $\endgroup$ – julieth Mar 14 '13 at 17:24
  • 4
    $\begingroup$ Quote from the Agresti book: "The separate-fitting estimates differ from the ML estimates for simultaneous fitting of the J-1 logits. They are less efficient, tending to have larger standard errors. However, Begg and Gray 1984 showed that the efficiency loss is minor when the response category having highest prevalence is the baseline.". $\endgroup$ – Franck Dernoncourt Oct 25 '15 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.