Proof for the standard error of parameters in linear regression

Question

In the book Introduction to Statistical Learning, the authors describe the relation between predictor $X$ and response $Y$, by linear regression as: $$ Y = \beta_{0} + \beta_{1}X+\epsilon$$
Here, $\beta_0$ is the intercept term and $\beta_1$ is the slope. $\epsilon$ is the error term.
By minimizing the least squares criterion, the values of $\hat{\beta_0}$ and $\hat{\beta_1}$ are found to be: $$\hat\beta_1 = \frac {\sum_{i=1}^{n}{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^{n}(x_i-\bar{x})^2}$$

$$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ Finding this wasn't very hard. Next, the authors find the standard error(SE) of these parameters. They do so by firstly providing the following : $$Var(\hat\mu)=SE(\hat\mu)^2=\frac{\sigma^2}{n}$$ That is, $SE = \frac {\sigma}{\sqrt{n}}$ (where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$).
Next, the authors give the standard errors of both the parameters: $$SE(\hat\beta_0)^2=\sigma^2 \Big[ \frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\Big]$$
$$SE(\hat\beta_1)^2=\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}$$ Where $\sigma^2=Var(\epsilon)$. There seems no connection between the formulas found for the parameters, and their standard errors. In order to find the standard error, we must have the variance of both the parameters. But how can we find the variance of formulas? They're bound to give an exact value for a particular input.

Answer 1

Note $Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2Var(\hat{\beta}_1) - 2Cov(\bar{y},\hat{\beta}_1)$. Try to show that the covariance term is 0.

The $Var(\hat{\mu}) = \dfrac{\sigma^2}{n}$ fact (although I'm not a fan of the notation they used here) is used in the calculation, $Var(\bar{y}) = \dfrac{\sigma^2}{n}$.

Answer 2

Proof for standard error of parameters: $$\begin{align*} \mathrm{SE}\left(\hat{\beta}_1\right)^2 &= \mathrm{Var}\left(\frac{\sum_i\left(x_i - \bar{x}\right)\left(y_i - \bar{y}\right)}{\sum_i\left(x_i - \bar{x}\right)^2}\right) \\ &= \mathrm{Var}\left(\frac{\sum_i(x_i - \bar{x})y_i}{\sum_i(x_i - \bar{x})^2}\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \mathrm{Var}\left(\sum_i\left(x_i - \bar{x}\right)y_i\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sum_i\left(x_i - \bar{x}\right)^2 \mathrm{Var}\left(y_i\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sum_i\left(x_i - \bar{x}\right)^2\sigma^2 \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sigma^2\sum_i\left(x_i - \bar{x}\right)^2 \\ &= \frac{\sigma^2}{\sum_i\left(x_i - \bar{x}\right)^2} \end{align*}$$

Answer 3

Once we have @ThorGirl's derivation for $\mathrm{SE}\left(\hat{\beta}_1\right)^2$ we can use that to derive the Standard Error for $\hat{\beta}_0$, i.e. $\mathrm{SE}\left(\hat{\beta}_0\right)^2 $.

Note: If you are looking for a step-by-step explanation of @ThorGirl's answer take a look at this video.

We are going to use the following assumptions / observations:

1. Each output $y_i$ is assumed to have the form: $\beta_0 + \beta_1 x_i + \epsilon_i$
2. $\overline{y} = \frac{1}{n}\sum_{i=1}^{n} y_i$
3. The error terms $\epsilon_i$ are uncorrelated with constant variance $\sigma^2$
4. $y_i$, $\hat{\beta}_0$ and $\hat{\beta}_1$ are random variables.
5. $x_i$ are known constants. $\beta_0$ and $\beta_1$ are the true values of the estimators, hence are also constants.

During the derivation we are going to use the following properties of variance. Here $a$ and $b$ are constants and $X$ and $Y$ are random variables.

1. $\mathrm{Var}(aX+b) = a^2\mathrm{Var}(X)$
2. $\mathrm{Var}(aX+bY) = a^2\mathrm{Var}(X) + b^2\mathrm{Var}(y) + 2ab \mathrm{Cov}\left(X,\:Y\right)$

We start with the definition of $\hat{\beta}_0$ : $$ \hat{\beta}_0 = \overline{y} - \hat{\beta}_1 \overline{x} $$

Therefore we have: $$ \mathrm{Var}\left(\overline{y} - \hat{\beta}_1 \overline{x}\right) $$

At this point if $\overline{y}$ and $\hat{\beta}_1$ are uncorrelated we can take each term's variance. This uncorrelation is what @Flowsnake's answer refers to. $$ \mathrm{Var}\left(\overline{y}\right) + \mathrm{Var}\left(-\hat{\beta}_1 \overline{x}\right) $$

As the predictors (inputs) $x_i$ are generally assumed to be known we can treat $\overline{x}$ as a constant. $$ \mathrm{Var}\left(\overline{y}\right) + \left(-\overline{x}\right)^2 \mathrm{Var}\left(\hat{\beta}_1\right) $$

Using @ThorGirl's answer, we can substitute $\mathrm{Var}\left(\hat{\beta}_1\right)$: $$ \mathrm{Var}\left(\overline{y}\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

The following steps will focus on $\mathrm{Var}\left(\overline{y}\right)$. As $y_i$ is not a constant, we must make a few substitutions. $$ \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^ny_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

By definition of the statistical model, we can further expand the term. $$ \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n\left(\beta_0 + \beta_1x_i + \epsilon_i\right)\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\mathrm{Var}\left(\sum_{i=1}^n\left(\beta_0 + \beta_1x_i + \epsilon_i\right)\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

As $\epsilon_i$ is uncorrelated, the variance of the sum of random variables becomes a sum of individual variances. $$ \frac{1}{n^2}\sum_{i=1}^n\mathrm{Var}\left(\beta_0 + \beta_1x_i + \epsilon_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\sum_{i=1}^n\mathrm{Var}\left(\epsilon_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\sum_{i=1}^n\sigma^2 + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n}\sigma^2 + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \sigma^2 \left[\frac{1}{n} + \frac{\overline{x}^2}{\sum_{i=1}^n\left(x_i - \overline{x}\right)^2}\right] $$