I have seen it claimed in Hosmer & Lemeshow (and elsewhere) that least squares parameter estimation in logistic regression is suboptimal (does not lead to a minimum variance unbiased estimator). Does anyone know other books/articles that show/discuss this explicitly? Google and my personal library have not helped me here...

  • $\begingroup$ Are you asking for a list of such books? It is common knowledge that OLS regression is inappropriate with a binary response variable, if you need a citation, H&L should be good enough. What is the point of this question? $\endgroup$ – gung - Reinstate Monica Mar 19 '12 at 21:59
  • 7
    $\begingroup$ Gung, As the OP says, H&L merely state that when LS is used, "the estimators no longer have [a number of desirable properties]" [2nd Ed. pp 5-6]. This statement is vague, it is offered without further explanation, and no citation is provided. This question does not solicit a list, but rather seeks credible support for an otherwise unfounded assertion. I think it's a fair question and gladly vote it up. $\endgroup$ – whuber Mar 19 '12 at 22:57
  • 1
    $\begingroup$ Gung: whuber stated most of what I would have stated. I also want to make it clear that I am not talking about fitting a regression model with normal error (to data with binary response). I am talking about using the usual logistic regression model, except estimating the parameters via least squares instead of via maximum likelihood methods. $\endgroup$ – user9968 Mar 20 '12 at 1:10
  • $\begingroup$ For independent binary outcomes, if the true mean response $\mathbb{E}[Y|X]$ follows a logistic-linear model, then optimal tests & estimates are indeed given by logistic regression. If the truth isn't logistic-linear, you don't the same uniform optimality. You do, however, get semi-parametric efficiency from logistic regression, among the class of linear estimation equations, following from Godambe and Heyde's theory of optimal EEs, exploiting the known mean-variance relationship for binary outcomes. LS estimation is worse in both cases - which doesn't make it universally worse. $\endgroup$ – guest Mar 20 '12 at 6:56
  • 3
    $\begingroup$ One reason why least squares is sub-optimal, is that for any binary random variable, $X$, we have $E[X^k]=E[X]=Pr(X=1)$ for all $k$. This means that the variance of $X$ is equal to $E[X](1-E[X])$. So you would at least need to do iterative weighted least squares, as the variance is not independent of the mean (no homoscedasticity). This is basically what logistic regression does, with the added feature that it forces all predictions to lie in $[0,1]$. $\endgroup$ – probabilityislogic Apr 4 '12 at 0:43

It is a well known fact that if the model is parametric (that is, specified completely up to a finite number of unknown parameters), and certain regularity conditions hold, then Maximum Likelihood estimation is asymptotically optimal (in the class of regular estimators). I have doubts about the UMVUE concept, since MLE rarely gives unbiased estimators.

The question of why is MLE optimal is rather tough, you can check for example Van der Vaart's "Asymptotic Statistics", chapter 8.

Now it is known that least squares coincides with MLE if and only if the distribution of error terms in the regression is normal (you can check the OLS article on Wikipedia). Since in logistic regression the distribution is not normal, LS will be less efficient than MLE.

  • $\begingroup$ I too doubt that UMVUE is relevant here. What you can compare though is asymptotic efficiency, and it will be lower for the (nonlinear) least squares for the reasons you explained. $\endgroup$ – StasK Apr 26 '12 at 2:38
  • $\begingroup$ Your last paragraph is not quite right. If you are talking about regression, it means that the goal of analysis is $E[y|x]$. Logistic regression, instead, is after the conditional probability ${\rm Pr}[y=1|x]$. It is, of course, a matter of coincidence that $E[y|x]={\rm Pr}[y=1|x]\times 1+ {\rm Pr}[y=0|x]\times0={\rm Pr}[y=1|x]$, so you can just as well run a nonlinear regression with $\Lambda(x'\beta)$ as the conditional mean function, and that's what H&L probably had in mind. $\endgroup$ – StasK Apr 26 '12 at 2:42

In ordinary linear regression maximizing the likelihood is equivalent to minimizing the sum of squared errors across the board (and consequently the estimated variance of errors) I In logistic regression, the errors are not expected to have the same variance: we should have high variance for p near .5, lower variance towards the extremes I Leads to (iteratively) (re)weighted least squares (IRWLS) method, where errors are penalized more where we expect less variance around p


Not the answer you're looking for? Browse other questions tagged or ask your own question.