How to get the standard error of linear regression parameters?

Question

I am reading through the textbook An introduction to Statistical learning with Applications in R, and got stuck:

When doing linear regression, we can write the relation as:

$$Y = \beta_0 + \beta_1X + \epsilon$$

where $$E(\epsilon)=0$$

And, we can define the residual sum of squares (RSS) as:

$$\text{RSS} = e_1^2 + e_2^2 + \dots +e_n^2,$$ where $e_i = y_i - \hat{y}_i$.

To minimize the RSS, we can get:

$$\hat{\beta}_1 = \dfrac{\sum\limits_{i = 1}^{n}(x_i - \overline{x})(y_i - \overline{y})}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}$$ $$\hat{\beta}_0 = \overline{y} - \hat{\beta}_1 \overline{x}$$

But, I cannot get the standard error with $\hat{\beta}_0$, which is (according to the book, but I don't know how to prove it):

$$\text{SE}(\hat{\beta_0})^2 = \text{Var}(\epsilon)\left[\dfrac{1}{n} + \dfrac{\overline{x}^2}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}\right]$$ $$\text{SE}(\hat{\beta}_1)^2 = \dfrac{\text{Var}(\epsilon)}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}$$

How to prove $\text{SE}(\hat{\beta}_0)^2$ and $\text{SE}(\hat{\beta}_1)^2$?

Answer 1

Hint:

Write $$\widehat{\beta_1}= \frac{\sum_{i=1}^n \left( x_i-\bar{x} \right)}{\sum_{i=1}^n \left(x_i-\bar{x} \right)^2} y_i $$

and you can check that these two expressions are equivalent, as the sum of mean deviations is zero. Since we are treating the predictors as fixed, you can use the properties of the variance to get what you want.

Now for the intercept, again using the standard rules of variance, we have

$$Var\left\{\widehat{\beta_0}\right\}=Var\left\{\bar{y} \right\}+\bar{x}^2 Var\left\{ \widehat{\beta}_1 \right\}-2\bar{x} Cov\left\{\bar{y},\widehat{\beta}_1 \right\}$$

But now note that

$$Cov\left\{\bar{y}. \widehat{\beta}_1\right\}=Cov\left\{\frac{1}{n}\sum_{i=1}^n y_i, \frac{\sum_{i=1}^n \left( x_i-\bar{x} \right)}{\sum_{i=1}^n \left(x_i-\bar{x} \right)^2} y_i \right\} $$

and since the $y_i$s are independent this reduces to zero. I leave to you the details of this computation. Thus the variance of $\widehat{\beta}_0$ is the sum of the first two components and if you plug in everything, you should get what you are looking for.

Hope this helps.

Answer 2

The long algebraic manipulations involved in standard demonstrations are bothersome. There ought to be a demonstration that is simple, direct, and provides insight into what the terms in the formulas mean.

By "simple" I will allow liberal use of substitution in formulas and application of the most straightforward algebraic reductions, along with solving linear equations in one unknown. Even square roots are to be avoided. I also count as simple the basic manipulations of variances and covariances that go under the concept of bilinearity. Sequences of simple manipulations that one can easily make mentally are placed together on single lines below. Neither variance calculation takes more than one line.

One way to accomplish this simplification employs a useful statistical principle: choose suitable units of measurement adapted to the data. Because we posit a specific way in which $y$ depends on $x,$ let's focus on how to measure $x.$

A good unit of measurement of the $x_i$ is one that standardizes them.

This means we will want to write

$$x_i = \sigma_x \xi_i + \bar x$$

where $\bar x$ is the usual arithmetic mean of the $x_i$ and $\sigma_x$ is their standard deviation (not an estimate!) defined by

$$\sigma_x^2 = \frac{1}{n}\sum_{i=1}^n \left(x_i - \bar x\right)^2.$$

In these units, the sum of the $\xi_i$ is zero and the sum of their squares is $n.$ The model is

$$y_i = \beta_0 + \beta_1 x_i \ =\ (\beta_0 + \beta_1 \bar x) + (\beta_1 \sigma_x)\xi_i + \varepsilon_i.\tag{1}$$

This says that in the new units the slope is $\beta_1 \sigma_x$ and the intercept is $\beta_0 + \beta_1 \bar x.$

Variance of the slope estimate

The Ordinary Least Squares estimate of the slope in these new units is simply the average product of the $\xi_i$ and $y_i,$

$$\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i.\tag{2}$$

To find its variance, look at the model $(1):$ the only parts of this that are random variables are the $\varepsilon_i$ terms. In $(2)$ they are multiplied by $\xi/n.$ Assuming these random variables are uncorrelated and each has a variance $\sigma^2,$ it is immediate that

$$\sigma_x^2 \operatorname{Var}\left(\hat\beta_1\right) = \operatorname{Var}\left(\widehat{\beta_1\sigma_x}\right) = \sum_{i=1}^n \operatorname{Var}\left(\frac{\xi_i}{n}y_i\right) = \sum_{i=1}^n \left(\frac{\xi_i}{n}\right)^2 \sigma^2 = \frac{1}{n}\sigma^2.$$

(The last simplification equated the sum of squares of the $\xi$ with $n,$ as noted before.) The solution is just as simple,

$$ \operatorname{Var}\left(\hat\beta_1\right) = \color{Red}{\frac{1}{\sigma_x^2}\left(\frac{1}{n}\sigma^2\right)} = \frac{\sigma^2}{\sum_{i=1}^n (x_i-\bar x)^2},$$

as claimed. However, it is the middle expression that is interpretable. It is the product of three factors:

1. The factor $\frac{1}{\sigma_x^2}$ is due to the units of measurement of the $x_i.$

2. The factor $\frac{1}{n}$ is the reduction in variance achieved by averaging $n$ uncorrelated random values in equation $(2).$

3. The factor $\sigma^2$ is the common variance of all those deviations.

Let us note in passing that the slope estimate is uncorrelated with the mean of the $y$ values. This follows from an easy calculation of the covariance,

$$\operatorname{Cov}\left(\hat\beta_1, \bar y\right) =\operatorname{Cov}\left(\sum_{i=1}^n \frac{\xi_i}{n\sigma_x} y_i, \frac{1}{n}\sum_{j=1}^n y_j\right) = \left(\frac{1}{n\sigma_x}\right)\left(\frac{1}{n}\right)\sum_{i=1}^n \xi_i \sigma^2 = 0.$$

The reduction from a double sum to a single sum is due to the zero correlation of $y_i$ and $y_j$ for $i\ne j$ (as well as the constant variance of the $\varepsilon_i$) and the final simplification is due to the sum-to-zero identity of the standardized variables $\xi_i.$

Variance of the intercept estimate

The estimated intercept when the explanatory variables $x_i$ are centered at zero is just the mean of the $y_i.$ Upon recentering--which subtracts $\bar x$ from all $x$ values--the estimate is thereby changed by $\hat\beta_1$ times $-\bar x.$ Thus,

$$\hat \beta_0 = \frac{1}{n}\sum_{i=1}^n y_i - \bar x\hat\beta_1.$$

Taking variances (and exploiting the zero correlation of the two terms) gives

$$\begin{aligned} \operatorname{Var}\left(\hat \beta_0\right) &= \operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n y_i\right) + \operatorname{Var}\left(-\bar x\hat\beta_1\right) = \color{Red}{\left(\frac{1}{n}\right)^2\left(n\sigma^2\right) + \left(-\bar x\right)^2 \operatorname{Var}\left(\hat\beta_1\right)}\\&= \sigma^2\left[\frac{1}{n} + \frac{\left(\bar x\right)^2}{n\sigma_x^2}\right]. \end{aligned}$$

The unsimplified result (on the top line) exposes a statistical interpretation:

1. The result, as before, is proportional to the common variance of all the error terms $\varepsilon_i.$

2. The proportionality is a sum of two pieces.

   • The $1/n$ piece comes from averaging the $y_i$ to estimate the intercept for standardized $x$ values.

   • The other piece accounts for shifting the $x$ values by the amount $-\bar x$ to center them. It, in turn, is the product of two quantities: the factor $(-\bar x)^2$ accounts for that shift (squared, because it appears in a variance) while the factor $1/(n\sigma_x^2)$ accounts for how the estimated slope translates that into a shift in $y.$

Answer 3

The principle for derivation below

The estimators $\beta_0$ and $\beta_1$ are linear estimators. That means that you can write it as a weighted sum of the observations $y_i$.

$$\hat\beta_ 0 = \sum b_{0i} y_i\\\hat\beta_ 1 = \sum b_{1i} y_i$$

As a consequence, we see that the sampling distribution of the estimators corresponds to the distribution of the sum of variables $y_i$. If we assume that the $y_i$ are independently distributed and with variance $\sigma^2$ then the variance of the estimators will be

$$\begin{array}{} \text{var}(\hat\beta_0) &=& \sum (b_{0i})^2 \text{var}\left(y_i\right) &=& \sigma^2 \sum (b_{0i})^2\\ \text{var}(\hat\beta_1) &=& \sum (b_{1i})^2 \text{var}\left(y_i\right) &=& \sigma^2 \sum (b_{1i})^2 \end{array}$$

Working it out

The $b_{0i}$ and $b_{1i}$ can be found by rewriting your formulas for the estimators.

$$\begin{array} \hat{\beta}_1 &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i - \overline{y})}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i) - \sum_{i = 1}^{n}(x_i - \overline{x})(\overline{y})}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i)}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=&\sum_{i = 1}^{n} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) (y_i)\\ \end{array} $$ and $$\begin{array} \hat{\beta}_0 &=& \overline{y} - \hat{\beta}_1 \overline{x}\\ &=& \frac{1}{n} \sum y_i - \sum_{i = 1}^{n} \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) (y_i)\\ &=& \sum_{i = 1}^{n} \left[\frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\right] (y_i)\\ \end{array} $$

Which gives

$$\begin{array}\\ b_{0i} &=& \frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\\ b_{1i} &=& \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \end{array}$$

If you take the sum of squares of these $b_{0i}$ and $b_{1i}$ you get

$$\begin{array}\\ \sum_{i=1}^n (b_{1i})^2 &=& \sum_{i=1}^n \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2\\ &=& \left( \dfrac{1}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \right)^2 \cdot \sum_{i=1}^n \left( x_i - \overline{x}\right)^2\\ &=& \dfrac{1}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \end{array}$$

and

$$\begin{array}\\ \sum_{i=1}^n (b_{0i})^2 &=& \sum_{i=1}^n \left[ \frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\right]^2\\ &=& \sum_{i=1}^n \left[ \frac{1}{n^2} + \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 - 2 \frac{1}{n} \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) \right]\\ &=& \sum_{i=1}^n \left[ \frac{1}{n^2} + \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 \right]\\ &=& \sum_{i=1}^n \frac{1}{n^2} + \sum_{i=1}^n \left[ \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 \right]\\ &=& \frac{1}{n} + \dfrac{\overline{x}^2}{ \sum_{j = 1}^{n}(x_j - \overline{x})^2 } \end{array}$$