# Simple Linear Regression - Unbiasedness

We consider the residuals $$\varepsilon_i$$ as random variables drawn independently from some distribution with mean zero. In other words, for each value of $$x$$, the corresponding value of $$y$$ is generated as a mean response $$\alpha + \beta x$$ plus an additional random variable $$\varepsilon$$ called the error term, equal to zero on average. Under such interpretation, the least-squares estimators $$\hat \alpha$$ and $$\hat \beta$$ will themselves be random variables whose means will equal the "true values" $$\alpha$$ and $$\beta$$.

Wikipedia

Is there a conceptual way to understand how is it that the normal distribution of the error (which is somewhat acceptable) turns $$\alpha$$ and $$\beta$$ into normally distributed?

• The quote does not say they are normally distributed. Nonetheless, they are normally distributed and the proof can be found here en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares
– Tim
Jun 25, 2023 at 11:21
• I have read that page, but I do not see any proof for that specific statement @Tim Jun 25, 2023 at 14:58
• Check the “Finite-sample distribution” section
– Tim
Jun 25, 2023 at 17:16
• As Tim and Dave pointed out, assuming that $\epsilon$ is normally distributed means that the parameters $\hat{\alpha},\hat{\beta}$ are also normal in finite-samples by a property of normal distributions (any affine transformation of a normal is normal). If $\epsilon$ is not normal, if the data is i.i.d. then the parameters are asymptotically normal as $n\to\infty$. This can be shown by a simple proof with the Central Limit Theorem.
Jun 25, 2023 at 22:51

For a general parameter vector $$\beta$$ and model matrix $$X$$, the OLS solution for a linear model is given by $$\hat\beta = (X^TX)^{-1}X^Ty$$. Assume that $$\varepsilon_i\overset{iid}{\sim}N(0, \sigma^2)$$.
By the assumptions of the model, $$y = X\beta + \varepsilon$$.
$$(X^TX)^{-1}X^Ty\\ =(X^TX)^{-1}X^T(X\beta + \varepsilon)\\ =(X^TX)^{-1}(X^TX)\beta + (X^TX)^{-1}X^T\varepsilon\\ =\beta + ((X^TX)^{-1}X^T)\varepsilon$$
A linear transformation of a multivariate normal vector, such as $$((X^TX)^{-1}X^T)\varepsilon$$, is multivariate normal. Thus, the entire $$\hat\beta$$ has a normal distribution with mean $$\beta$$ $$($$shift the normal $$((X^TX)^{-1}X^T)\varepsilon$$ by $$\beta$$, which will, again, be normal$$)$$. That the mean is $$\beta$$ gives the unbiasedness.
Simple linear regression is a special case of this, where the parameter vector $$\beta$$ just has a slope and an intercept parameter, often called $$\beta$$ and $$\alpha$$, respectively, and the model matrix is a column of $$1$$s next to a column of the observations of the explanators variable, $$x$$.