2
$\begingroup$

The Maximum Likelihood estimator theory comes often with the the theoretical result on the variance:

$\sigma^2(\hat \theta_{MLE}) \sim -\left(E[\frac{\partial^2 log L(\theta=\theta_0)}{\partial^2 \theta}]\right)^{-1}$

, called the inverse of the Fisher information ($\theta_0$ are the unknown exact parameters). The proofs I saw of this result always consider the problem of estimating the parameters of an unknown distribution having at disposal $\{y_1,...y_T\}$, a sample of identically distributed and independent extractions. These kind of proofs to me do not seem to apply immediately to simple linear regression, where we try to fit the model:

$\hat y_i = \alpha x_i +\hat \epsilon_i$

and the sample here is extracted from variables $\{y_1,...y_T\}$ which are independent but not identically distributed (e.g. $E[\hat y_i]=\alpha x_i$). Nevertheless, MLE theory is used to derive formulas of linear regression and also their variance.

Where can I find a presentation of MLE which applies directly to linear regression problems? (or maybe understand why the standard presentation applies...)

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ OLS assumption: $\hat \epsilon_i$ i.i.d. $N(0,\sigma^2)$. Now blast away with MLE. $\endgroup$
    – Mark L. Stone
    Commented Jan 14, 2017 at 12:51
  • $\begingroup$ Thanks, but is it not a problem, to make the connection, that the errors $\hat \epsilon_i$ are unobserved and the parameter $\alpha$ unknown (so that we cannot infere the error therm) ? $\endgroup$
    – Thomas
    Commented Jan 14, 2017 at 13:40
  • $\begingroup$ It is not a problem. The errors are unknown. If you knew what the errors were, they wouldn't need to be errors. The distribution of the errors is assumed known, other than the unknown parameter $\sigma^2$. $\endgroup$
    – Mark L. Stone
    Commented Jan 14, 2017 at 21:06
  • $\begingroup$ I make an example to explain my doubt. In this pdf: le.ac.uk/users/dsgp1/COURSES/THIRDMET/MYLECTURES/1XMAXILIKE.pdf a fundamental point to get Fisher estimate of the variance is equation (22). It relies on the central limit theorem and the fact that the likelyhood can be written as $L=\prod_{i=1}^N f(y_i; \theta)$, function f being fixed. How can this framework applied to OLS ? In that case the likelyhood is of the type $L=\prod_{i=1}^N f(y_i; \mu_i ,\theta)$ with the non random parameters $\mu_i$ depending on $i$.Ido not get how the central limit theorem applied in the second one. $\endgroup$
    – Thomas
    Commented Jan 16, 2017 at 14:54
  • $\begingroup$ MLE has "nothing" to do with Central Limit Theorem, unless the asymptotic behavior is considered. Anyhow, look at stats.stackexchange.com/questions/143705/… and the links in there. $\endgroup$
    – Mark L. Stone
    Commented Jan 16, 2017 at 16:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.