Using MSE instead of log-loss in logistic regression

Question

Suppose we replace the loss function of the logistic regression (which is normally log-likelihood) with the MSE. That is, still have log odds ratio be a linear function of the parameters, but minimize the sum of squared differences between the estimated probability and the outcome (coded as 0 / 1):

$\log \frac p{1-p} = \beta_0 + \beta_1x_1 + ... +\beta_nx_n$

and minimize $\sum(y_i - p_i)^2$ instead of $\sum [y_i \log p_i + (1-y_i) \log (1-p_i)]$.

Of course, I understand why log likelihood makes sense under some assumptions. But in machine learning, where assumptions are usually not made, what is the intuitive reason the MSE is completely unreasonable? (Or are there situations where MSE might make sense?).

Answer 1

The short answer is that likelihood theory exists to guide us towards optimum solutions, and maximizing something other than the likelihood, penalized likelihood, or Bayesian posterior density results in suboptimal estimators. Secondly, minimizing sum of squared errors leads to unbiased estimates of true probabilities. Here you do not desire unbiased estimates, because to have that estimates can be negative or greater than one. To properly constrain estimates requires one to get slightly biased estimates (towards the middle) in general, on the probability (not the logit) scale.

Don't believe that machine learning methods do not make assumptions. This issue has little to do with machine learning.

Note that an individual proportion is an unbiased estimate of the true probability, hence a binary logistic model with only an intercept provides an unbiased estimate. A binary logistic model with a single predictor that has $k$ mutually exclusive categories will provide $k$ unbiased estimates of probabilities. I think that a model that capitalizes on additivity assumptions and allows the user to request estimates outside the data range (e.g., a single predictor that is continuous) will have a small bias on the probability scale so as to respect the $[0,1]$ constraint.

Answer 2

Although Frank Harrell's answer is correct, I think it misses the scope of the question. The answer to your question is yes, MSE would make sense in a ML nonparametric scenario. The ML equivalent of logistic regression is the linear perceptron, which makes no assumptions and does use MSE as a cost function. It uses online gradient descent for parameter training and, since it solves a convex optimisation problem, parameter estimates should be at the global optimum. The main difference between the two methods is that with the nonparametric approach you don't get confidence intervals and p-values and therefore you cannot use your model for inference, you can only use it for prediction.

The Linear Perceptron makes no probabilistic assumptions. There is the assumption on the data that it is linearly separable, but this is not an assumption on the model. MSE could be in theory affected by heteroscedasticity but in practice this effect is nullified by the activation function.