53
$\begingroup$

Lets say we have a dependent variable $Y$ with few categories and set of independent variables.

What are the advantages of multinomial logistic regression over set of binary logistic regressions (i.e. one-vs-rest scheme)? By set of binary logistic regression I mean that for each category $y_{i} \in Y$ we build separate binary logistic regression model with target=1 when $Y=y_{i}$ and 0 otherwise.

$\endgroup$
4
  • 4
    $\begingroup$ Mathematically, a multinomial logit model is a set of binary logit models, all compared against a base alternative. But because you get to collapse generic parameters and maybe combine some others, the MNL will always be at least as efficient (and probably more so). I see no reason to ever use a series of binomial models. $\endgroup$ Mar 14 '13 at 11:52
  • 3
    $\begingroup$ @gmacfarlane: I've tried to simulate data where MNL would be better than series of binary logistic regressions, but every time on average the quality was the same. I was comparing lift charts and after averaging results from few simulations they looke almost the same. Maybe You have an idea how to generate data so MNL beats binary logistic regressions? Although MNL had a great advantage, its scores could be interpreted as probability. $\endgroup$ Mar 14 '13 at 14:16
  • $\begingroup$ Multinomial Logistic regression is the extension of binary logit regression. It is used when the dependent variables of the study is three and above, whereas, binary logit is used when the dependent variables of the study is two. $\endgroup$
    – user48573
    Jun 18 '14 at 12:20
  • 1
    $\begingroup$ To reader: I recommend starting at @julieth's answer and following up by reading ttnphns'. I think the former more directly answers the original question but the latter adds some interesting context. ttnphns also shows the different features that are available for both in a popular software routine, which could itself constitute a reason to use one over the other (see gregmacfarlane's statement). $\endgroup$
    – Ben Ogorek
    Feb 16 '19 at 16:42
31
$\begingroup$

If $Y$ has more than two categories your question about "advantage" of one regression over the other is probably meaningless if you aim to compare the models' parameters, because the models will be fundamentally different:

$\bf log \frac{P(i)}{P(not~i)}=logit_i=linear~combination$ for each $i$ binary logistic regression, and

$\bf log \frac{P(i)}{P(r)}=logit_i=linear~combination$ for each $i$ category in multiple logistic regression, $r$ being the chosen reference category ($i \ne r$).

However, if your aim is only to predict probability of each category $i$ either approach is justified, albeit they may give different probability estimates. The formula to estimate a probability is generic:

$\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+exp(logit_j)+\dots+exp(logit_r)}$, where $i,j,\dots,r$ are all the categories, and if $r$ was chosen to be the reference one its $\bf exp(logit)=1$. So, for binary logistic that same formula becomes $\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+1}$. Multinomial logistic relies on the (not always realistic) assumption of independence of irrelevant alternatives whereas a series of binary logistic predictions does not.


A separate theme is what are technical differences between multinomial and binary logistic regressions in case when $Y$ is dichotomous. Will there be any difference in results? Most of the time in the absence of covariates the results will be the same, still, there are differences in the algorithms and in output options. Let me just quote SPSS Help about that issue in SPSS:

Binary logistic regression models can be fitted using either the Logistic Regression procedure or the Multinomial Logistic Regression procedure. Each procedure has options not available in the other. An important theoretical distinction is that the Logistic Regression procedure produces all predictions, residuals, influence statistics, and goodness-of-fit tests using data at the individual case level, regardless of how the data are entered and whether or not the number of covariate patterns is smaller than the total number of cases, while the Multinomial Logistic Regression procedure internally aggregates cases to form subpopulations with identical covariate patterns for the predictors, producing predictions, residuals, and goodness-of-fit tests based on these subpopulations. If all predictors are categorical or any continuous predictors take on only a limited number of values—so that there are several cases at each distinct covariate pattern—the subpopulation approach can produce valid goodness-of-fit tests and informative residuals, while the individual case level approach cannot.

Logistic Regression provides the following unique features:

  • Hosmer-Lemeshow test of goodness of fit for the model
  • Stepwise analyses
  • Contrasts to define model parameterization
  • Alternative cut points for classification
  • Classification plots
  • Model fitted on one set of cases to a held-out set of cases
  • Saves predictions, residuals, and influence statistics

Multinomial Logistic Regression provides the following unique features:

  • Pearson and deviance chi-square tests for goodness of fit of the model
  • Specification of subpopulations for grouping of data for goodness-of-fit tests
  • Listing of counts, predicted counts, and residuals by subpopulations
  • Correction of variance estimates for over-dispersion
  • Covariance matrix of the parameter estimates
  • Tests of linear combinations of parameters
  • Explicit specification of nested models
  • Fit 1-1 matched conditional logistic regression models using differenced variables
$\endgroup$
5
  • $\begingroup$ I know that these models will be different, but I don't know which one is better in which situation. I will ask the question in another way. If You were given a task: For each person predict the probability that some mobile phone company is the favourite one (lets assume every one have favourite mobile phone company). Which of those methods would You use and what are the advantages over the second one? $\endgroup$ Mar 14 '13 at 9:40
  • $\begingroup$ @Tomek I expanded my answer a little bit $\endgroup$
    – ttnphns
    Mar 14 '13 at 17:31
  • $\begingroup$ Though I think @julieth's is the best answer to O.P.'s original question, I do owe you for the introduction to the Independence of Irrelevant Alternatives assumption. One question I still have is whether separate logistics truly get around it; the Wikipedia article you linked to mentioned probit and "nested logit" as allowing violations of IIA $\endgroup$
    – Ben Ogorek
    Feb 16 '19 at 16:36
  • $\begingroup$ Would you be able to explain how to fit the models with a choice of reference category? For category $i$, do we only use a subset of the data that is either in the reference category $r$ or the category $i$, for $i \neq r$? $\endgroup$
    – user21359
    May 6 '19 at 21:19
  • $\begingroup$ @ttnphns can you please elaborate how you got from assumption that exp(logit) = 1 for reference variable to the fact that probability from multinomial reduces to one from binomial regression? Do you mean to say sum of exp(logit) for all categories from j to r (excluding i) is equal to 1? $\endgroup$
    – toing
    Jan 25 at 19:26
22
$\begingroup$

Because of the title, I'm assuming that "advantages of multiple logistic regression" means "multinomial regression". There are often advantages when the model is fit simultaneously. This particular situation is described in Agresti (Categorical Data Analysis, 2002) pg 273. In sum (paraphrasing Agresti), you expect the estimates from a joint model to be different than a stratified model. The separate logistic models tend to have larger standard errors although it may not be so bad when the most frequent level of the outcome is set as the reference level.

$\endgroup$
4
  • $\begingroup$ Thanks! I will try to find this book, unfortunatelly google.books provides content only till page 268. $\endgroup$ Mar 14 '13 at 13:39
  • $\begingroup$ @TomekTarczynski I summarized the relevant information from the paragraph, so you may not get any more info related to this question from looking at the book (although the book is great so you will get other good info). $\endgroup$
    – julieth
    Mar 14 '13 at 17:24
  • 9
    $\begingroup$ Quote from the Agresti book: "The separate-fitting estimates differ from the ML estimates for simultaneous fitting of the J-1 logits. They are less efficient, tending to have larger standard errors. However, Begg and Gray 1984 showed that the efficiency loss is minor when the response category having highest prevalence is the baseline.". $\endgroup$ Oct 25 '15 at 16:16
  • $\begingroup$ @TomekTarczynski take a look here: libgen.is/… $\endgroup$ Aug 17 '20 at 23:07
4
$\begingroup$

I don't think the previous answers really capture the key difference, although it is implicit in the discussion of Independence of Irrelevant Alternatives (which is a social sciences term rather than a statistical term).

If you use a multinomial model then your predictions for the different options sum to 1; If you use n different logistic regression models they won't.

The multinomial model is to be preferred when there is a fixed set of classes, and they are mutually exclusive.

So for instance in case: "For each person predict the probability that some mobile phone company is the favourite one (lets assume every one have favourite mobile phone company). Which of those methods would You use and what are the advantages over the second one?"

If you believe there is a fixed unchanging set of phone companies, then multinomial regression would be appropriate. If instead you are eg predicting top 3 (which are fixed), but there is also a tail of smaller companies you don't model, then I would suggest 1 vs rest for top 3 companies is appropriate (because top 3 doesn't cover 100% of respondents)

$\endgroup$
1
$\begingroup$

It seems that the question was not at all about the implementation/structural differences between (a) the softmax (multinomial logistic) regression model and (b) the OvR "composite" model based on multiple binary logistic regression models. In a nutshell, however, skipping all the formulas, these differences can be summarized like this:

  • Training: the softmax regression model uses the cross entropy cost function, while the OvR "composite" model based on multiple binary logistic regressors trains completely independent binary logit classifiers using the logistic regression cost function.
  • Trained model representation: not much difference - in softmax each class gets its own parameter vector, and these vectors are stored together in a common parameter matrix, while in OvR logit there are exactly as many separate parameter vectors, one for each positive class.
  • Evaluation: the softmax regression model uses the softmax function that predicts a probability for each class considering the scores for other classes, while the OvR "composite" model based on multiple binary logistic regressors calculates the scores/probabilities of classes completely independently and then just picks the label with the highest score.

It also seems that there was no need to explain the differences between the binary, the OvR/OvO "composite" models and the "native" multilabel classifiers like the multinomial logistic regressor (aka the softmax regressor).

I think the question was more about THE ACCURACY:

The softmax regression (LogisticRegression(multi_class="multinomial") in scikit-learn) is more flexible when setting the linear decision boundaries among the classes. Here is a two-dimensional three-class illustration of this:
https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_multinomial.html

The above example really could have benefitted from the confusion matrices, so here they are (normalized):

enter image description here

This is not a hard classification problem - the instances from the three classes are barely mixed, so we should expect very high accuracy for all the classes. But OvR Logit stumbles when identifying the "middle" class. Generally speaking, OvR Logit will perform poorly when there is low distinction for some class by the feature values alone. It only likes "edgy" classes.

For binary classification this is not a disadvantage when compared to Softmax/multinomial, since the latter also sets a linear boundary between the two classes.
Or imagine three clusters that are at approximately the same distances from each other (i.e. each class cluster is on the vertex of an equilateral triangle). In such a case the accuracy of both OvR Logit and Softmax will be good for all the classes.

However, imagine one of the three clusters being at or near the straight line between the centers of the other two clusters... The accuracy of OvR Logit for that "middle" class will be poor. Softmax/multinomial regressor will do well (even though its decision boundaries are still straight lines).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.