1
$\begingroup$

I recently read about logistic regression model. $$ y=\frac{1}{1+e^{-(\beta_0+\beta_1x)}}. $$ What seems to be perplexing to me is, I can see different optimization tools being used to estimate the parameters $\beta_0$ and $\beta_1$.

Here, maximum likelihood based estimation technique has been used. Whereas we could have simply used the least square based technique to estimate the parameter.

Can you please tell which one is the best method to estimate parameters for logistic regression and why?

$\endgroup$
4
  • $\begingroup$ @Dave, right. I was posting an answer for a different version of the question. It was about MLE vs. SGD. I wonder why I cannot find it in the version history of the post. I was quite sure it involved SGD, but I cannot find that! Or perhaps I am just mistaken. $\endgroup$ Commented Jan 31, 2022 at 10:13
  • 3
    $\begingroup$ Like weighted least squares, MLE implicitly weights observations differently, giving more accurate parameter estimates. Try using a simulation where the response probabilities range from very small to values around .5 for your X values, and compare accuracy of OLS vs MLE over 1000s of replications. $\endgroup$ Commented Jan 31, 2022 at 10:30
  • 1
    $\begingroup$ Check out the first answer to this related question stats.stackexchange.com/questions/326350/… . While least-squares is not a good optimisation algorithm for LR, iteratively reweighted least-squares works pretty well ( en.wikipedia.org/wiki/Iteratively_reweighted_least_squares ) $\endgroup$ Commented Apr 12 at 6:45
  • 1
    $\begingroup$ Because the conditional variance of $y$ depends on the coefficients $\beta_i,$ at a minimum you would want to use suitably weighted least squares. That can work pretty well, actually--and it gives results identical to usual logistic regression when iteratively applied to use previous estimates for the weights until convergence. $\endgroup$
    – whuber
    Commented Apr 12 at 16:42

2 Answers 2

1
$\begingroup$

Least squares estimates, when the outcome variable Y=0, 1, will result in some predicted probabilities that are negative or greater than 1.0. As these are illegal values, this is not good performance to say the least. Maximum likelihood estimates (of which least squares is an example only for normally distributed continuous Y) are guaranteed to be efficient as $N \rightarrow \infty$. Not only will MLEs be efficient but they respect the structure of the problem by restricting estimated probabilities to be in $[0,1]$.

So think of it this way: Always use MLEs (or penalized MLEs or Bayesian estimates which are also based on the likelihood function). In the case of conditionally normally distributed Y (normal residuals) the MLEs happen to be the least squares estimates. The equivalence in the special Gaussian case stems from the fact that the log likelihood function is proportional to the sum of squared errors plus a constant.

$\endgroup$
8
  • 2
    $\begingroup$ I took the question to mean minimizing Brier score. While that would not be an MLE, all predictions would be legal. $\endgroup$
    – Dave
    Commented Jan 31, 2022 at 13:42
  • $\begingroup$ Oh yes. In that case you are right they would all be legal, just not statistically efficient. $\endgroup$ Commented Jan 31, 2022 at 13:46
  • 1
    $\begingroup$ Many reasons - narrower confidence limits, lower mean squared error of predicted probabilities, ability to add penalty terms to the log likelihood, ability to add a prior to the log-likelihood to do a Bayesian analysis, extension to longitudinal data, extension to multinomial and ordinal data, formal hypothesis tests. $\endgroup$ Commented Jan 31, 2022 at 15:50
  • 4
    $\begingroup$ This answers a different question. The one that was asked is whether "least squares methods" can be used to estimate the parameters in a logistic regression model. It is impossible that any such estimates would produce probabilities outside the $[0,1]$ interval. $\endgroup$
    – whuber
    Commented Apr 12 at 15:40
  • 1
    $\begingroup$ I see your point @whuber - least squares on binary data will give invalid estimates but is not using a logistic model form. $\endgroup$ Commented Apr 14 at 12:44
2
$\begingroup$

If your dataset is $\{(x_i, y_i)|i=1..n\}$, where $y_i\in[0,1]$, then using head-on least squares, like $$ \sum_i (\hat{y}(x_i)-y_i)^2\longrightarrow \min$$ is a legitimate procedure... but it will require numerical optimization. So it is not easier than maximum likelihood... but statistically inferior.

After logit transformation one obtains $$ \log\left(\frac{y}{1-y}\right)=-\beta_0-\beta_1x, $$ which makes it look mathematically as amenable to simple linear regression for the transoformed variable... except that the log is not defined for values $y=0,1$.

$\endgroup$
4
  • 1
    $\begingroup$ Why do you need numerical optimization instead of using the usual $(X^TX)^{-1}X^Ty?$ $\endgroup$
    – Dave
    Commented Apr 12 at 15:14
  • $\begingroup$ Because $y$ is usually binary in these regressions, your logit transformation isn't even defined. Thus, regardless of what values $y$ might have, that proposal must be invalid. $\endgroup$
    – whuber
    Commented Apr 12 at 15:41
  • $\begingroup$ You could assume a small level of label noise and use targets of e.g. 0.000001 and 0.99999 to get around the log not being defined. It would make little difference to the standard LR model in most cases (and how often are we certain one in a million patterns are not mislabelled). I'm not sure it would work very well though. $\endgroup$ Commented Apr 12 at 15:42
  • 1
    $\begingroup$ ... thinking about it a bit more, it wouldn't matter how much label noise was used, it would just end up being LS regression to arbitrary fixed values for each class, so it would be equivalent (up to a rescaling) to Fisher's linear discriminant. However, this isn't what the question was asking. $\endgroup$ Commented Apr 12 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.