3
$\begingroup$

For a linear regression problem $y=X\beta + \epsilon$, I think we know very well that the estimated $\hat{\beta} = \dfrac{X^Ty}{X^TX}$ is unbiased, and has the variance introduced by $\epsilon$.

It sounds reasonable to me that over the years, we might have a good understanding of this same question for logistic regression also, but I cannot find any.

I wonder if we have these studies for Logistic regression, or maybe it's not even possible to study these questions because Logistic regression does not have a closed-form solution of $\beta$?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ Check the MLE (maximum likelihood estimate). Estimate of regression coefficients in logistic regression is MLE, so the properties of MLE is applicable for logistic regression. $\endgroup$
    – user158565
    Commented Oct 19, 2018 at 15:34

1 Answer 1

Reset to default
3
$\begingroup$

Answers by Alecos Papadopoulos on this site show two distinct ways in which logistic regression coefficients based on maximum likelihood estimation (MLE) are biased.

This page shows a closed-form calculation based on a simple illustrative situation. Although the probability estimates themselves are unbiased in this situation, the MLE estimates of coefficients are biased for finite sample sizes.

An additional difference between ordinary linear regression and logistic regression is the potential contribution of omitted-variable bias in the two situations. Unlike in ordinary linear regression, omitting a predictor associated with outcome in logistic regression necessarily leads to bias toward 0 in the regression coefficients of the included predictors even if the omitted predictor is uncorrelated with the included predictors. Some discussion and a nice closed-form derivation for the related case of a probit model is on this page.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, but there is still one thing I'm confused: When we talk about the variance of estimated $\beta$, it's usually just number. Therefore, we can calculate the MSE as squared bias plus variance, but in Alecos's answer, the variance derived is a matrix. Could you please help me understand what happened? $\endgroup$
    – user42004
    Commented Oct 19, 2018 at 21:30
  • 2
    $\begingroup$ @HaohanWang the estimated variance values for the individual coefficients are those along the diagonal of the covariance matrix. If you want to calculate the variance of linear combinations of the coefficients (e.g., contrasts), then you also need to know the off-diagonal covariances. That's also true for covariance matrices in ordinary linear regression. Writing out derivations in matrix form is typically most efficient; introductory texts and courses often omit matrices, which can seem daunting until one has a fair amount of experience with them. $\endgroup$
    – EdM
    Commented Oct 19, 2018 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.