Logistic regression: maximum likelihood vs misclassification

Question

Traditionally the fitting of the logistic regression function is explained using maximum likelihood. Could one fit the logistic regression function as well based on either least-squares or based on misclassification error, or is this not possible / has drawbacks?

Answer 1

One could possible estimate the logistic regression model by minimizing the classification error, but there is usually no reason why to do so! Why do you want to do it?

But, such questions have been asked here before, so I will not rewrite an answer, very good answers can be found to this question: Logistic regression: maximizing true positives - false positives

Basically, minimizing misclassification error amounts to using a score function which is not a proper scoring rule, see: https://en.wikipedia.org/wiki/Scoring_rule If misclassification is minimized by some parameter vector $\beta$, it will be also minimized by many other values of $\beta$ in some vicinity of the first $\beta$. In other words, the criterion function is flat around the maximum! To see this last fact, we explore the connection with scoring rules (se wiki above). We specialize to the case with only a binary variable, possible values 0 or 1, with distribution given by a probability vector $p=[p_1, p_2]$. Let the random variable be $X$, the forecaster makes a probabilistic forecast $r=[r_1,r_2]$, a probability vector. Let $S$ be a score function.

This means that if the forecaster forecasts $r$, then $X=x$ is observed, he receives the reward $S(r,x)$. This reward then has expected value $E S(r,X)=p_1 S(r,1)+p_2 S(r,2)$ and we say the scoring rule (reward) is proper if this expectation is maximized (under $p$) by forecasting $r=p$. It is strictly proper if that maximum is unique.

Trying to minimize the misclassification rate corresponds to using the following score function: $$ S(r,i)=\begin{cases} 1 ~~\text{if $p_i=\max(p_1,p_2)$} \\ 0 ~~\text{otherwise} \end{cases} $$ $(i=1,2)$ Now, consider if the true $p=[0.99,0.01]$. If the forecaster then reports $r=p=[0.99, 0.01]$ then his expected reward becomes $$ E S(r,X) = p_1 S(r,1)+ p_2 S(r,2) = p_1 $$ Consider then if the forecaster report $r=s$, some other probability vector such that $s_1 > s_2$, that is $s_1 > 0.5$. Then you can calculate as above, and find the exact same expected reward! So this score function fails to reward truthful forecasting, it only depends on the event actually forecasted having been given a probability larger than one half.

That is the reason use of this will not lead to very effective learning! and so should be avoided.

Answer 2

In a word, yes, but it wouldn't be logistic regression anymore.

The logistic regression loss function (i.e., the negative log likelihood), is essentially a regression in the log odds. Changing that to a least squares loss function will make it linear regression, which will lose three things: (i) the interpretation of the regression coefficients in terms of log odds and (ii) interpretation of the model predictions as log odds (or when exponentiated, as probabilities) (iii) that linear regression does not bound the model predictions between 0 and 1, so it can easily make predictions <0 or >1. That said, for many real world applications, simply minimizing the least squares criteria using a 0/1 response (output, DV) works quite well, because there enough noise etc so that the model never gets close to 0 or 1 (the sigmoid function around 0.5 well approximated by a straight line). Plus in terms of things like ROC performance, that doesn't matter since all you care about is the ranks of the predictions.

For the misclassification error, that loss function is not differentiable and not convex (also called 0/1 loss), so it's very hard to minimize effectively. Hence, both support vector machines and logistic regression minimize two convex proxy loss functions, the hinge loss and the logistic loss, respectively, which can be seen as approximations to the 0/1 loss (convex relaxations).

Answer 3

I believe you can reduce the probability of misclassification by introducing a class of random effects mixing distributions. Using this approach, you can develop a full-likelihood model, which includes effects of misclassification [Ref 1, 2, 3].

However, there are two major challenges in applying this modelling approach:

• Relaxation of the assumption on the mixing distribution of the nuisance parameters
• Determining the efficiency and reliability of the model's validations

Finally, you might like to look at the semi-parametric Maximum Likelihood algorithms, such as [Ref 4].

---

1. Rice, K. M. (2004). Equivalence between conditional and mixture approaches to the Rasch model and matched case-control studies, with applications. Journal of the American Statistical Association, 99(466), 510-522.
2. Roeder, K., Carroll, R. J., & Lindsay, B. G. (1996). A semiparametric mixture approach to case-control studies with errors in covariables. Journal of the American Statistical Association, 91(434), 722-732.
3. Rice, K. (2003). Full‐likelihood approaches to misclassification of a binary exposure in matched case‐control studies. Statistics in medicine, 22(20), 3177-3194.
4. Schafer, D. W. (2001). Semiparametric maximum likelihood for measurement error model regression. Biometrics, 57(1), 53-61.