6
$\begingroup$

This question already has an answer here:

I have a doubt on why sum of squared errors is not used for Logistic regression and instead maximum likelihood estimation is used and also why not the vice versa.

Edited

Many were asking me to clarify the question, my intention was to know about using a squared error loss function to find β vs MLE

$\endgroup$

marked as duplicate by kjetil b halvorsen, user158565, Michael Chernick, whuber Jul 18 at 0:42

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ Your observations are of the form $0$ and $1$. If you tried to apply the function $f(x)=\log\left(\frac{x}{1-x}\right)$ to these, you might have difficulties $\endgroup$ – Henry Dec 14 '16 at 7:50
  • $\begingroup$ Sir, can you be more specific about the difficulties $\endgroup$ – Sri Harsha Pinninti Dec 14 '16 at 10:52
  • 6
    $\begingroup$ If you insist: neither $\log\left(\frac{0}{1}\right)$ nor $\log\left(\frac{1}{0}\right)$ are finite, which makes makes linear regression of the transformed probabilities difficult. You could instead try non-linear regression, but then you lose the arithmetic advantages of linear regression and you should consider carefully your loss function: using maximum likelihood rewards accurate prediction of high and low probability events $\endgroup$ – Henry Dec 14 '16 at 12:40
  • 1
    $\begingroup$ Minimizing squared errors is a maximum likelihood estimation -- of the wrong likelihood function. $\endgroup$ – jwimberley Jan 1 '17 at 21:59
  • $\begingroup$ @Henry: I don't support this idea, but you could keep the response unchanged (i.e. $y = 0, 1$) and have the predictions be $\hat y = \frac{e^\eta}{1 + e^\eta}$ with $\eta = X^T \beta$, and finally minimize $\sum (y_i - \hat y_i)^2$. I believe this is what the OP intended in the question. $\endgroup$ – Cliff AB Jan 3 '17 at 1:19
2
$\begingroup$

Firstly, least squares (or sum of squared errors) is a possible loss function to use to fit your coefficients. There's nothing technically wrong about it.

However there are number of reasons why MLE is a more attractive option. In addition to those in the comments, here are two more:

Computational efficiency

Because the likelihood function of a logistic regression model is a member of the exponential family, we can use Fisher's Scoring algorithm to efficiently solve for $\beta$. In my experience, this algorithm converges in only a few steps. To solve least squares numerically will likely take longer.

Lest this gets lost, per @vbox's comment:

learning parameters for any machine learning model (such as logistic regression) is much easier if the cost function is convex. And, it's not too difficult to show that, for logistic regression, the cost function for the sum of squared errors is not convex, while the cost function for the log-likelihood is.

MLE has very nice properties

Solutions using MLE have nice properties such:

  • consistency: meaning that with more data, our estimate of $\beta$ gets closer to the true value.
  • asymptotic normality: meaning that with more data, our estimate of $\beta$ is approximately normal distributed with variance that decreases with $O(\frac{1}{n})$
  • functional invariance: nice property to have when dealing with multiple parameters (nuisance parameters) and calculating the profile likelihood.

Among others.

However using Least Squares does have some benefits

Least squares tends to be more robust to outliers because an outlier can be wrong by at most 1 (because $(1-0)^2 = 1$), whereas under a negative log likelihood loss function, the distance can be arbitrarily large.

For more information check this or this out.

Edited

My interpretation of the OPs question is why do we use MLE instead of a square loss function to determine $\beta$ in a logistic regression model of the form:

$$logit(P(Y=1|X)) = x\beta$$

Where $P(Y=1|X) = f(x;\beta) = \frac{e^{x\beta}}{1 + e^{x\beta}} = \frac{1}{1 + e^{-x\beta}}$

So the loss function looks like:

$$\sum_{i} (y_i - f(x_i;\beta))^2 = \sum_{i} (y_i - \frac{1}{1 + e^{-x\beta}})^2$$

where $y_i$'s take values 0/1.

When I talk about computational efficiency, I mean finding the $\beta$ which minimizes the above vs. Fisher Scoring on the likelihood function.

$\endgroup$
  • $\begingroup$ OLS is generally more computationally efficient than logistic regression; a single step of Fisher's scoring should have more computational costs than solving the entire OLS solution. $\endgroup$ – Cliff AB Jan 3 '17 at 0:04
  • $\begingroup$ Is this true for least squares with a logistic function? Can you point me to a reference? $\endgroup$ – ilanman Jan 3 '17 at 0:13
  • $\begingroup$ Ah, so you mean something like a GLM with a logistic link and Gaussian pdf. In that case, it would not be in closed form like OLS, but I am extremely doubtful it will be faster. $\endgroup$ – Cliff AB Jan 3 '17 at 1:09
  • 4
    $\begingroup$ To add to ilanman's answer, learning parameters for any machine learning model (such as logistic regression) is much easier if the cost function is convex. And, it's not too difficult to show that, for logistic regression, the cost function for the sum of squared errors is not convex, while the cost function for the log-likelihood is. $\endgroup$ – vbox Jan 3 '17 at 2:47
  • 1
    $\begingroup$ @vbox: ah, I forget about the non-convexity issue. In that case, I could see how using the logistic transform with a squared error loss could be slower. Sorry for the distraction. $\endgroup$ – Cliff AB Jan 3 '17 at 17:20
0
$\begingroup$

Maybe I'm not getting the point of ilanman's answer as well as of some of the comments here, but afaiks, the answer is simply that

OLS = log L(Gaussian)

i.e. the OLS corresponds to the log likelihood of a regression with a normal / Gaussian distribution. You can see this by logging the formula for the Gaussian - the $\sigma$ will factor out, and you see that the OLS maximizes the likelihood.

So, OLS estimation IS MLE for a Gaussian error.

Logistic Regression assumes a Bernoulli / Binomial error, that is why you don't do OLS.

$\endgroup$
  • $\begingroup$ I think the OP is asking about using a squared error loss function to find $\beta$ vs MLE. $\endgroup$ – ilanman Jan 3 '17 at 12:10
  • $\begingroup$ Hmm ... maybe it would be useful if the OP could clarify what the question was. In my world, the main use of logistic regression is 0/1 data, and there you would like to fit a glm with logit link / binomial distribution - OLS simply doesn't fit to that assumptions. $\endgroup$ – Florian Hartig Jan 4 '17 at 11:17
  • $\begingroup$ I agree, but technically you could fit a squared loss function. And to be fair, no where does the OP use the term OLS. $\endgroup$ – ilanman Jan 4 '17 at 11:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.