1
$\begingroup$

We consider the residuals $\varepsilon_i$ as random variables drawn independently from some distribution with mean zero. In other words, for each value of $x$, the corresponding value of $y$ is generated as a mean response $\alpha + \beta x$ plus an additional random variable $\varepsilon$ called the error term, equal to zero on average. Under such interpretation, the least-squares estimators $\hat \alpha$ and $\hat \beta$ will themselves be random variables whose means will equal the "true values" $\alpha$ and $\beta$.

Wikipedia

Is there a conceptual way to understand how is it that the normal distribution of the error (which is somewhat acceptable) turns $\alpha$ and $\beta$ into normally distributed?

$\endgroup$
4
  • 1
    $\begingroup$ The quote does not say they are normally distributed. Nonetheless, they are normally distributed and the proof can be found here en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares $\endgroup$
    – Tim
    Jun 25, 2023 at 11:21
  • $\begingroup$ I have read that page, but I do not see any proof for that specific statement @Tim $\endgroup$
    – Minsky
    Jun 25, 2023 at 14:58
  • $\begingroup$ Check the “Finite-sample distribution” section $\endgroup$
    – Tim
    Jun 25, 2023 at 17:16
  • $\begingroup$ As Tim and Dave pointed out, assuming that $\epsilon$ is normally distributed means that the parameters $\hat{\alpha},\hat{\beta}$ are also normal in finite-samples by a property of normal distributions (any affine transformation of a normal is normal). If $\epsilon$ is not normal, if the data is i.i.d. then the parameters are asymptotically normal as $n\to\infty$. This can be shown by a simple proof with the Central Limit Theorem. $\endgroup$
    – Adam
    Jun 25, 2023 at 22:51

1 Answer 1

2
$\begingroup$

For a general parameter vector $\beta$ and model matrix $X$, the OLS solution for a linear model is given by $\hat\beta = (X^TX)^{-1}X^Ty$. Assume that $\varepsilon_i\overset{iid}{\sim}N(0, \sigma^2)$.

By the assumptions of the model, $y = X\beta + \varepsilon$.

$$ (X^TX)^{-1}X^Ty\\ =(X^TX)^{-1}X^T(X\beta + \varepsilon)\\ =(X^TX)^{-1}(X^TX)\beta + (X^TX)^{-1}X^T\varepsilon\\ =\beta + ((X^TX)^{-1}X^T)\varepsilon $$

A linear transformation of a multivariate normal vector, such as $((X^TX)^{-1}X^T)\varepsilon$, is multivariate normal. Thus, the entire $\hat\beta$ has a normal distribution with mean $\beta$ $($shift the normal $((X^TX)^{-1}X^T)\varepsilon$ by $\beta$, which will, again, be normal$)$. That the mean is $\beta$ gives the unbiasedness.

Simple linear regression is a special case of this, where the parameter vector $\beta$ just has a slope and an intercept parameter, often called $\beta$ and $\alpha$, respectively, and the model matrix is a column of $1$s next to a column of the observations of the explanators variable, $x$.

(This is all taken from the "Finite-sample distribution" section of the Wikipedia article that Tim mentioned, and it should be found as a derivation or an exercise in a linear regression textbook.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.