6
$\begingroup$

I am reading through the textbook An introduction to Statistical learning with Applications in R, and got stuck:

When doing linear regression, we can write the relation as:

$$Y = \beta_0 + \beta_1X + \epsilon$$

where $$E(\epsilon)=0$$

And, we can define the residual sum of squares (RSS) as:

$$\text{RSS} = e_1^2 + e_2^2 + \dots +e_n^2,$$ where $e_i = y_i - \hat{y}_i$.

To minimize the RSS, we can get:

$$\hat{\beta}_1 = \dfrac{\sum\limits_{i = 1}^{n}(x_i - \overline{x})(y_i - \overline{y})}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}$$ $$\hat{\beta}_0 = \overline{y} - \hat{\beta}_1 \overline{x}$$

But, I cannot get the standard error with $\hat{\beta}_0$, which is (according to the book, but I don't know how to prove it):

$$\text{SE}(\hat{\beta_0})^2 = \text{Var}(\epsilon)\left[\dfrac{1}{n} + \dfrac{\overline{x}^2}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}\right]$$ $$\text{SE}(\hat{\beta}_1)^2 = \dfrac{\text{Var}(\epsilon)}{\sum\limits_{i = 1}^{n}(x_i - \overline{x})^2}$$

How to prove $\text{SE}(\hat{\beta}_0)^2$ and $\text{SE}(\hat{\beta}_1)^2$?

$\endgroup$
5
  • 1
    $\begingroup$ What about a self-study tag ;) ? $\endgroup$
    – RUser4512
    Sep 21, 2015 at 11:42
  • $\begingroup$ Keep in mind that both $\beta_0$ and $\beta_1$ are random variables and the standard errors are the square roots of the variances of these two random variables. $\endgroup$
    – xjf
    Sep 21, 2015 at 14:15
  • $\begingroup$ My (now closed) related question is here stats.stackexchange.com/q/545255/163242 . This is what I am seeking the answer to with my bounty. $\endgroup$ Sep 20, 2021 at 15:33
  • $\begingroup$ @ThePointer You didn't need to spend any points on a bounty: this is a FAQ. See stats.stackexchange.com/… for many answers. $\endgroup$
    – whuber
    Sep 20, 2021 at 16:44
  • $\begingroup$ @whuber I searched through several of those, and none of them contain answers that answer my question. I think this is bounty points well spent, if it finally results in a clear, detailed, step-by-step answer to this common question. $\endgroup$ Sep 21, 2021 at 10:48

3 Answers 3

2
$\begingroup$

Hint:

Write $$\widehat{\beta_1}= \frac{\sum_{i=1}^n \left( x_i-\bar{x} \right)}{\sum_{i=1}^n \left(x_i-\bar{x} \right)^2} y_i $$

and you can check that these two expressions are equivalent, as the sum of mean deviations is zero. Since we are treating the predictors as fixed, you can use the properties of the variance to get what you want.

Now for the intercept, again using the standard rules of variance, we have

$$Var\left\{\widehat{\beta_0}\right\}=Var\left\{\bar{y} \right\}+\bar{x}^2 Var\left\{ \widehat{\beta}_1 \right\}-2\bar{x} Cov\left\{\bar{y},\widehat{\beta}_1 \right\}$$

But now note that

$$Cov\left\{\bar{y}. \widehat{\beta}_1\right\}=Cov\left\{\frac{1}{n}\sum_{i=1}^n y_i, \frac{\sum_{i=1}^n \left( x_i-\bar{x} \right)}{\sum_{i=1}^n \left(x_i-\bar{x} \right)^2} y_i \right\} $$

and since the $y_i$s are independent this reduces to zero. I leave to you the details of this computation. Thus the variance of $\widehat{\beta}_0$ is the sum of the first two components and if you plug in everything, you should get what you are looking for.

Hope this helps.

$\endgroup$
3
  • $\begingroup$ I'm downvoting this because I don't think it does well in answering the question – which is probably why it's had no upvotes after six years, despite the number of views (and it being accepted as the answer). $\endgroup$ Sep 20, 2021 at 15:10
  • $\begingroup$ @ThePointer You are wrong, the answer gives the OP what he's looking for in the form of hints. Everything is there. $\endgroup$
    – JohnK
    Sep 20, 2021 at 19:31
  • $\begingroup$ I think it's there, too (+1) and, moreover, conforms with our policy for answering routine ("self-study") questions, where we prefer to offer guidance and hints. After time passes we tend to relax that policy and supply full answers, but that should in no way deprecate the original answer(s) that were offered. $\endgroup$
    – whuber
    Sep 21, 2021 at 16:03
4
+50
$\begingroup$

The long algebraic manipulations involved in standard demonstrations are bothersome. There ought to be a demonstration that is simple, direct, and provides insight into what the terms in the formulas mean.

By "simple" I will allow liberal use of substitution in formulas and application of the most straightforward algebraic reductions, along with solving linear equations in one unknown. Even square roots are to be avoided. I also count as simple the basic manipulations of variances and covariances that go under the concept of bilinearity. Sequences of simple manipulations that one can easily make mentally are placed together on single lines below. Neither variance calculation takes more than one line.

One way to accomplish this simplification employs a useful statistical principle: choose suitable units of measurement adapted to the data. Because we posit a specific way in which $y$ depends on $x,$ let's focus on how to measure $x.$

A good unit of measurement of the $x_i$ is one that standardizes them.

This means we will want to write

$$x_i = \sigma_x \xi_i + \bar x$$

where $\bar x$ is the usual arithmetic mean of the $x_i$ and $\sigma_x$ is their standard deviation (not an estimate!) defined by

$$\sigma_x^2 = \frac{1}{n}\sum_{i=1}^n \left(x_i - \bar x\right)^2.$$

In these units, the sum of the $\xi_i$ is zero and the sum of their squares is $n.$ The model is

$$y_i = \beta_0 + \beta_1 x_i \ =\ (\beta_0 + \beta_1 \bar x) + (\beta_1 \sigma_x)\xi_i + \varepsilon_i.\tag{1}$$

This says that in the new units the slope is $\beta_1 \sigma_x$ and the intercept is $\beta_0 + \beta_1 \bar x.$

Variance of the slope estimate

The Ordinary Least Squares estimate of the slope in these new units is simply the average product of the $\xi_i$ and $y_i,$

$$\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i.\tag{2}$$

To find its variance, look at the model $(1):$ the only parts of this that are random variables are the $\varepsilon_i$ terms. In $(2)$ they are multiplied by $\xi/n.$ Assuming these random variables are uncorrelated and each has a variance $\sigma^2,$ it is immediate that

$$\sigma_x^2 \operatorname{Var}\left(\hat\beta_1\right) = \operatorname{Var}\left(\widehat{\beta_1\sigma_x}\right) = \sum_{i=1}^n \operatorname{Var}\left(\frac{\xi_i}{n}y_i\right) = \sum_{i=1}^n \left(\frac{\xi_i}{n}\right)^2 \sigma^2 = \frac{1}{n}\sigma^2.$$

(The last simplification equated the sum of squares of the $\xi$ with $n,$ as noted before.) The solution is just as simple,

$$ \operatorname{Var}\left(\hat\beta_1\right) = \color{Red}{\frac{1}{\sigma_x^2}\left(\frac{1}{n}\sigma^2\right)} = \frac{\sigma^2}{\sum_{i=1}^n (x_i-\bar x)^2},$$

as claimed. However, it is the middle expression that is interpretable. It is the product of three factors:

  1. The factor $\frac{1}{\sigma_x^2}$ is due to the units of measurement of the $x_i.$

  2. The factor $\frac{1}{n}$ is the reduction in variance achieved by averaging $n$ uncorrelated random values in equation $(2).$

  3. The factor $\sigma^2$ is the common variance of all those deviations.

Let us note in passing that the slope estimate is uncorrelated with the mean of the $y$ values. This follows from an easy calculation of the covariance,

$$\operatorname{Cov}\left(\hat\beta_1, \bar y\right) =\operatorname{Cov}\left(\sum_{i=1}^n \frac{\xi_i}{n\sigma_x} y_i, \frac{1}{n}\sum_{j=1}^n y_j\right) = \left(\frac{1}{n\sigma_x}\right)\left(\frac{1}{n}\right)\sum_{i=1}^n \xi_i \sigma^2 = 0.$$

The reduction from a double sum to a single sum is due to the zero correlation of $y_i$ and $y_j$ for $i\ne j$ (as well as the constant variance of the $\varepsilon_i$) and the final simplification is due to the sum-to-zero identity of the standardized variables $\xi_i.$

Variance of the intercept estimate

The estimated intercept when the explanatory variables $x_i$ are centered at zero is just the mean of the $y_i.$ Upon recentering--which subtracts $\bar x$ from all $x$ values--the estimate is thereby changed by $\hat\beta_1$ times $-\bar x.$ Thus,

$$\hat \beta_0 = \frac{1}{n}\sum_{i=1}^n y_i - \bar x\hat\beta_1.$$

Taking variances (and exploiting the zero correlation of the two terms) gives

$$\begin{aligned} \operatorname{Var}\left(\hat \beta_0\right) &= \operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n y_i\right) + \operatorname{Var}\left(-\bar x\hat\beta_1\right) = \color{Red}{\left(\frac{1}{n}\right)^2\left(n\sigma^2\right) + \left(-\bar x\right)^2 \operatorname{Var}\left(\hat\beta_1\right)}\\&= \sigma^2\left[\frac{1}{n} + \frac{\left(\bar x\right)^2}{n\sigma_x^2}\right]. \end{aligned}$$

The unsimplified result (on the top line) exposes a statistical interpretation:

  1. The result, as before, is proportional to the common variance of all the error terms $\varepsilon_i.$

  2. The proportionality is a sum of two pieces.

    • The $1/n$ piece comes from averaging the $y_i$ to estimate the intercept for standardized $x$ values.

    • The other piece accounts for shifting the $x$ values by the amount $-\bar x$ to center them. It, in turn, is the product of two quantities: the factor $(-\bar x)^2$ accounts for that shift (squared, because it appears in a variance) while the factor $1/(n\sigma_x^2)$ accounts for how the estimated slope translates that into a shift in $y.$

$\endgroup$
7
  • $\begingroup$ "The Ordinary Least Squares estimate of the slope in these new units is simply the average product of the $\xi_i$ and $y_i,$ $$\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i.\tag{2}$$" You state this without any justification whatsoever, which seems to be what you said you would avoid at the beginning. The reasoning cannot be followed beyond this point without justifying why "the Ordinary Least Squares estimate of the slope in these new units is simply the average product of the $\xi_i$ and $y_i$." $\endgroup$ Oct 30, 2021 at 19:46
  • 1
    $\begingroup$ @ThePointer This is, perhaps, the most basic and intuitive explanation of OLS as laid out, for instance, in any edition of Freedman, Pisani, & Purves, Statistics. You can also find this explanation and this formula by searching our site. You can also find it in discussions of the relationship between the correlation coefficient and the OLS slope. The question of justification is an interesting one. Drawing the line is a matter of keeping communication simple. For instance, do we also need to justify all the algebraic rules of arithmetic when performing calculations? $\endgroup$
    – whuber
    Oct 30, 2021 at 19:49
  • $\begingroup$ I searched en.wikipedia.org/wiki/Ordinary_least_squares, but I cannot see where this justification is explained. I assumed it was a more complex assumption/theorem that was used, but I could be wrong and it could be something simple that I am missing. $\endgroup$ Oct 30, 2021 at 19:51
  • $\begingroup$ @ThePointer You can find one account of this at stats.stackexchange.com/a/513296/919, another at stats.stackexchange.com/a/71303/919, and a simple example at stats.stackexchange.com/a/275021/919. Ultimately the explanation in Freeman et al. is the best, because they define the OLS solution in terms of the mean product of standardized variables. Some advanced regression texts do the same thing in their early motivational chapters. $\endgroup$
    – whuber
    Oct 30, 2021 at 19:59
  • $\begingroup$ @ThePointer I don't see where limits appear at all! They don't have anything to do with the questions or answers in this thread. The formula I give in my answer comes from the Normal equations for OLS regression. It is standard. $\endgroup$
    – whuber
    Oct 30, 2021 at 21:04
1
$\begingroup$

The principle for derivation below

The estimators $\beta_0$ and $\beta_1$ are linear estimators. That means that you can write it as a weighted sum of the observations $y_i$.

$$\hat\beta_ 0 = \sum b_{0i} y_i\\\hat\beta_ 1 = \sum b_{1i} y_i$$

As a consequence, we see that the sampling distribution of the estimators corresponds to the distribution of the sum of variables $y_i$. If we assume that the $y_i$ are independently distributed and with variance $\sigma^2$ then the variance of the estimators will be

$$\begin{array}{} \text{var}(\hat\beta_0) &=& \sum (b_{0i})^2 \text{var}\left(y_i\right) &=& \sigma^2 \sum (b_{0i})^2\\ \text{var}(\hat\beta_1) &=& \sum (b_{1i})^2 \text{var}\left(y_i\right) &=& \sigma^2 \sum (b_{1i})^2 \end{array}$$

Working it out

The $b_{0i}$ and $b_{1i}$ can be found by rewriting your formulas for the estimators.

$$\begin{array} \hat{\beta}_1 &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i - \overline{y})}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i) - \sum_{i = 1}^{n}(x_i - \overline{x})(\overline{y})}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=& \dfrac{\sum_{i = 1}^{n}(x_i - \overline{x})(y_i)}{\sum_{i = 1}^{n}(x_i - \overline{x})^2}\\ &=&\sum_{i = 1}^{n} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) (y_i)\\ \end{array} $$ and $$\begin{array} \hat{\beta}_0 &=& \overline{y} - \hat{\beta}_1 \overline{x}\\ &=& \frac{1}{n} \sum y_i - \sum_{i = 1}^{n} \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) (y_i)\\ &=& \sum_{i = 1}^{n} \left[\frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\right] (y_i)\\ \end{array} $$

Which gives

$$\begin{array}\\ b_{0i} &=& \frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\\ b_{1i} &=& \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \end{array}$$

If you take the sum of squares of these $b_{0i}$ and $b_{1i}$ you get

$$\begin{array}\\ \sum_{i=1}^n (b_{1i})^2 &=& \sum_{i=1}^n \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2\\ &=& \left( \dfrac{1}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \right)^2 \cdot \sum_{i=1}^n \left( x_i - \overline{x}\right)^2\\ &=& \dfrac{1}{\sum_{j = 1}^{n}(x_j - \overline{x})^2} \end{array}$$

and

$$\begin{array}\\ \sum_{i=1}^n (b_{0i})^2 &=& \sum_{i=1}^n \left[ \frac{1}{n} - \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)\right]^2\\ &=& \sum_{i=1}^n \left[ \frac{1}{n^2} + \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 - 2 \frac{1}{n} \overline{x} \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right) \right]\\ &=& \sum_{i=1}^n \left[ \frac{1}{n^2} + \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 \right]\\ &=& \sum_{i=1}^n \frac{1}{n^2} + \sum_{i=1}^n \left[ \overline{x}^2 \left( \dfrac{(x_i - \overline{x})}{\sum_{j = 1}^{n}(x_j - \overline{x})^2}\right)^2 \right]\\ &=& \frac{1}{n} + \dfrac{\overline{x}^2}{ \sum_{j = 1}^{n}(x_j - \overline{x})^2 } \end{array}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.