Lets say we have a dependent variable $Y$ with few categories and set of independent variables.
What are the advantages of multinomial logistic regression over set of binary logistic regressions (i.e. one-vs-rest scheme)? By set of binary logistic regression I mean that for each category $y_{i} \in Y$ we build separate binary logistic regression model with target=1 when $Y=y_{i}$ and 0 otherwise.
If $Y$ has more than two categories your question about "advantage" of one regression over the other is probably meaningless if you aim to compare the models' parameters, because the models will be fundamentally different:
$\bf log \frac{P(i)}{P(not~i)}=logit_i=linear~combination$ for each $i$ binary logistic regression, and
$\bf log \frac{P(i)}{P(r)}=logit_i=linear~combination$ for each $i$ category in multiple logistic regression, $r$ being the chosen reference category ($i \ne r$).
However, if your aim is only to predict probability of each category $i$ either approach is justified, albeit they may give different probability estimates. The formula to estimate a probability is generic:
$\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+exp(logit_j)+\dots+exp(logit_r)}$, where $i,j,\dots,r$ are all the categories, and if $r$ was chosen to be the reference one its $\bf exp(logit)=1$. So, for binary logistic that same formula becomes $\bf P'(i)= \frac{exp(logit_i)}{exp(logit_i)+1}$. Multinomial logistic relies on the (not always realistic) assumption of independence of irrelevant alternatives whereas a series of binary logistic predictions does not.
A separate theme is what are technical differences between multinomial and binary logistic regressions in case when $Y$ is dichotomous. Will there be any difference in results? Most of the time in the absence of covariates the results will be the same, still, there are differences in the algorithms and in output options. Let me just quote SPSS Help about that issue in SPSS:
Binary logistic regression models can be fitted using either the Logistic Regression procedure or the Multinomial Logistic Regression procedure. Each procedure has options not available in the other. An important theoretical distinction is that the Logistic Regression procedure produces all predictions, residuals, influence statistics, and goodness-of-fit tests using data at the individual case level, regardless of how the data are entered and whether or not the number of covariate patterns is smaller than the total number of cases, while the Multinomial Logistic Regression procedure internally aggregates cases to form subpopulations with identical covariate patterns for the predictors, producing predictions, residuals, and goodness-of-fit tests based on these subpopulations. If all predictors are categorical or any continuous predictors take on only a limited number of values—so that there are several cases at each distinct covariate pattern—the subpopulation approach can produce valid goodness-of-fit tests and informative residuals, while the individual case level approach cannot.
Logistic Regression provides the following unique features:
- Hosmer-Lemeshow test of goodness of fit for the model
- Stepwise analyses
- Contrasts to define model parameterization
- Alternative cut points for classification
- Classification plots
- Model fitted on one set of cases to a held-out set of cases
- Saves predictions, residuals, and influence statistics
Multinomial Logistic Regression provides the following unique features:
- Pearson and deviance chi-square tests for goodness of fit of the model
- Specification of subpopulations for grouping of data for goodness-of-fit tests
- Listing of counts, predicted counts, and residuals by subpopulations
- Correction of variance estimates for over-dispersion
- Covariance matrix of the parameter estimates
- Tests of linear combinations of parameters
- Explicit specification of nested models
- Fit 1-1 matched conditional logistic regression models using differenced variables
Because of the title, I'm assuming that "advantages of multiple logistic regression" means "multinomial regression". There are often advantages when the model is fit simultaneously. This particular situation is described in Agresti (Categorical Data Analysis, 2002) pg 273. In sum (paraphrasing Agresti), you expect the estimates from a joint model to be different than a stratified model. The separate logistic models tend to have larger standard errors although it may not be so bad when the most frequent level of the outcome is set as the reference level.
I don't think the previous answers really capture the key difference, although it is implicit in the discussion of Independence of Irrelevant Alternatives (which is a social sciences term rather than a statistical term).
If you use a multinomial model then your predictions for the different options sum to 1; If you use n different logistic regression models they won't.
The multinomial model is to be preferred when there is a fixed set of classes, and they are mutually exclusive.
So for instance in case: "For each person predict the probability that some mobile phone company is the favourite one (lets assume every one have favourite mobile phone company). Which of those methods would You use and what are the advantages over the second one?"
If you believe there is a fixed unchanging set of phone companies, then multinomial regression would be appropriate. If instead you are eg predicting top 3 (which are fixed), but there is also a tail of smaller companies you don't model, then I would suggest 1 vs rest for top 3 companies is appropriate (because top 3 doesn't cover 100% of respondents)
It seems that the question was not at all about the implementation/structural differences between (a) the softmax (multinomial logistic) regression model and (b) the OvR "composite" model based on multiple binary logistic regression models. In a nutshell, however, skipping all the formulas, these differences can be summarized like this:
It also seems that there was no need to explain the differences between the binary, the OvR/OvO "composite" models and the "native" multilabel classifiers like the multinomial logistic regressor (aka the softmax regressor).
I think the question was more about THE ACCURACY:
The softmax regression (LogisticRegression(multi_class="multinomial") in scikit-learn) is more flexible when setting the linear decision boundaries among the classes. Here is a two-dimensional three-class illustration of this:
https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_multinomial.html
The above example really could have benefitted from the confusion matrices, so here they are (normalized):
This is not a hard classification problem - the instances from the three classes are barely mixed, so we should expect very high accuracy for all the classes. But OvR Logit stumbles when identifying the "middle" class. Generally speaking, OvR Logit will perform poorly when there is low distinction for some class by the feature values alone. It only likes "edgy" classes.
For binary classification this is not a disadvantage when compared to Softmax/multinomial, since the latter also sets a linear boundary between the two classes.
Or imagine three clusters that are at approximately the same distances from each other (i.e. each class cluster is on the vertex of an equilateral triangle). In such a case the accuracy of both OvR Logit and Softmax will be good for all the classes.
However, imagine one of the three clusters being at or near the straight line between the centers of the other two clusters... The accuracy of OvR Logit for that "middle" class will be poor. Softmax/multinomial regressor will do well (even though its decision boundaries are still straight lines).
