4
$\begingroup$

A nice property of maximum likelihood estimators is that, while they can be biased, they are consistent for $iid$ observations.

In a logistic regression, unless the conditional distributions all have the same probability parameter, we lose the "identically" distributed of $iid$. Nonetheless, logistic regressions tend to have their parameters estimated by maximizing the likelihood.

(Why) does this not lead to an inconsistent maximum likelihood estimator?

Related post: Logistic regression panel data fixed effects.

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ That's what I thought, too, but evidently not. @JohnMadden $\endgroup$
    – Dave
    Commented Jan 30, 2023 at 23:23
  • 1
    $\begingroup$ "i.i.d." is a sufficient condition, but not a necessary condition for consistency. $\endgroup$
    – Zhanxiong
    Commented Jan 31, 2023 at 2:20
  • 1
    $\begingroup$ @Zhanxiong But why is there consistency despite the lack of $iid?$ $\endgroup$
    – Dave
    Commented Jan 31, 2023 at 2:28
  • 2
    $\begingroup$ @Dave Whereas a rigorous discussion requires a careful listing of assumptions on the parameter space, a quick answer is that: the MLE is a type of $M$-estimate, and the log-likelihood function of the logistic regression (under the usual regularity conditions) satisfies the condition of the Wald's consistency theorem (see Theorem 5.14 in Asymptotic Statistcs by A. W. van der Vaart), thus $\hat{\beta}$ is consistent. $\endgroup$
    – Zhanxiong
    Commented Jan 31, 2023 at 2:52
  • 2
    $\begingroup$ Dave, I added your other question's link in this post for the commentary there is definitely relevant here. $\endgroup$
    – User1865345
    Commented Jan 31, 2023 at 4:40

1 Answer 1

Reset to default
2
$\begingroup$

Some conditions on the explanatory variables are required for consistency

As with other regression models, the MLEs in logistic regression are not necessarily consistent --- consistency requires imposing some assumptions on the sequence of explanatory variables used in the limit. In the case of Gaussian linear regression we obtain consistency of the parameter estimator using the Grenander conditions (see related answers here, here and here), which (roughly speaking) requires the maximum leverage of the explanatory variables to converge to zero. It is likely that a similar requirement would be needed to prove consistency for the case of the logistic regression. In general, so long as the maximum leverage converges to zero you get a situation where no individual data point, or finite set of data points, is "influential" in the limit.

In view of this, the real question you need to be asking is: what is a reasonable set of sufficient conditions we can impose on the explanatory variables which yield a consistent MLE for the parameters in the logistic regression model? If you undertake this inquiry and find sufficient conditions of this kind, you will be able to see how consistency is established (usually in combination with asymptotic normality) and you will therefore be able to see why the required conditions are weaker than requiring the response variables in the model to be IID. While there are lots of different sufficient conditions that can be formulated, I recommend starting off by reviewing some standard conditions (see e.g., Fahrmeir and Kaufmann (1985) to get you started).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.