15
$\begingroup$

In the book Introduction to Statistical Learning, the authors describe the relation between predictor $X$ and response $Y$, by linear regression as: $$ Y = \beta_{0} + \beta_{1}X+\epsilon$$
Here, $\beta_0$ is the intercept term and $\beta_1$ is the slope. $\epsilon$ is the error term.
By minimizing the least squares criterion, the values of $\hat{\beta_0}$ and $\hat{\beta_1}$ are found to be: $$\hat\beta_1 = \frac {\sum_{i=1}^{n}{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^{n}(x_i-\bar{x})^2}$$

$$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ Finding this wasn't very hard. Next, the authors find the standard error(SE) of these parameters. They do so by firstly providing the following : $$Var(\hat\mu)=SE(\hat\mu)^2=\frac{\sigma^2}{n}$$ That is, $SE = \frac {\sigma}{\sqrt{n}}$ (where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$).
Next, the authors give the standard errors of both the parameters: $$SE(\hat\beta_0)^2=\sigma^2 \Big[ \frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\Big]$$
$$SE(\hat\beta_1)^2=\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}$$ Where $\sigma^2=Var(\epsilon)$. There seems no connection between the formulas found for the parameters, and their standard errors. In order to find the standard error, we must have the variance of both the parameters. But how can we find the variance of formulas? They're bound to give an exact value for a particular input.

$\endgroup$
6
  • 1
    $\begingroup$ Where did the parameter $\mu$ come from? $\endgroup$ Jul 8, 2017 at 3:00
  • 1
    $\begingroup$ (According to the book) $\mu$ is the population mean of a random variable $Y$ $\endgroup$
    – Mooncrater
    Jul 8, 2017 at 3:01
  • 2
    $\begingroup$ Your statement "In order to find the standard error, we must have the standard deviation of both the parameters" suggests a possible misunderstanding on your part, or perhaps two: 1. The parameters are fixed (but unknown) quantities and have no standard deviation. 2. The standard deviation of the distribution of the parameter estimates is called the standard error of the parameters. We can estimate that standard error from the regression. $\endgroup$
    – Glen_b
    Jul 8, 2017 at 11:28
  • $\begingroup$ @Glen_b Actually, by that line I wanted to put an emphasis on the fact that $$SE(\hat\mu)^2=var(\hat\mu)$$, which relates standard deviation and standard error. Meanwhile, I did have that doubt #1 in my head earlier. $\endgroup$
    – Mooncrater
    Jul 8, 2017 at 15:51
  • $\begingroup$ That actually relates the standard error of the mean to the variance of the sampling distribution of the estimate of the mean. $\endgroup$
    – Glen_b
    Jul 8, 2017 at 15:54

3 Answers 3

9
$\begingroup$

Note $Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2Var(\hat{\beta}_1) - 2Cov(\bar{y},\hat{\beta}_1)$. Try to show that the covariance term is 0.

The $Var(\hat{\mu}) = \dfrac{\sigma^2}{n}$ fact (although I'm not a fan of the notation they used here) is used in the calculation, $Var(\bar{y}) = \dfrac{\sigma^2}{n}$.

$\endgroup$
3
  • $\begingroup$ Due to my lack of experience and knowledge, I am unable to understand, how $cov(\bar{y},\hat{\beta_1})$ is $0$ (as, this is my first encounter with covariance). And after that, we still would need another relation between $var(\hat{\beta_0})$ and $var(\hat{\beta_1})$. That too, is too shy to be found by me. Sorry for my ignorance. $\endgroup$
    – Mooncrater
    Jul 8, 2017 at 10:05
  • 3
    $\begingroup$ @Mooncrater why is $Cov(\bar{y}, \hat{\beta}_{1}) = 0$? Hint: recall $\bar{y}$ is a constant. The formula for covariance of two r.v. $X$ and $Y$ is $Cov(X,Y) = E[XY] - E[X]E[Y]$. $\endgroup$ Jun 22, 2018 at 3:21
  • $\begingroup$ Could you also derive $Var(\hat{\beta}_1)$ $\endgroup$ Jun 26, 2019 at 1:27
13
$\begingroup$

Proof for standard error of parameters: $$\begin{align*} \mathrm{SE}\left(\hat{\beta}_1\right)^2 &= \mathrm{Var}\left(\frac{\sum_i\left(x_i - \bar{x}\right)\left(y_i - \bar{y}\right)}{\sum_i\left(x_i - \bar{x}\right)^2}\right) \\ &= \mathrm{Var}\left(\frac{\sum_i(x_i - \bar{x})y_i}{\sum_i(x_i - \bar{x})^2}\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \mathrm{Var}\left(\sum_i\left(x_i - \bar{x}\right)y_i\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sum_i\left(x_i - \bar{x}\right)^2 \mathrm{Var}\left(y_i\right) \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sum_i\left(x_i - \bar{x}\right)^2\sigma^2 \\ &= \frac{1}{\left(\sum_i\left(x_i - \bar{x}\right)^2\right)^2} \sigma^2\sum_i\left(x_i - \bar{x}\right)^2 \\ &= \frac{\sigma^2}{\sum_i\left(x_i - \bar{x}\right)^2} \end{align*}$$

$\endgroup$
1
  • 2
    $\begingroup$ If anyone are looking for a step-by-step explanation of @ThorGirl's answer take a look at this video. $\endgroup$
    – Hakan Baba
    Mar 11, 2020 at 7:23
7
$\begingroup$

Once we have @ThorGirl's derivation for $\mathrm{SE}\left(\hat{\beta}_1\right)^2$ we can use that to derive the Standard Error for $\hat{\beta}_0$, i.e. $\mathrm{SE}\left(\hat{\beta}_0\right)^2 $.

Note: If you are looking for a step-by-step explanation of @ThorGirl's answer take a look at this video.

We are going to use the following assumptions / observations:

  1. Each output $y_i$ is assumed to have the form: $\beta_0 + \beta_1 x_i + \epsilon_i$
  2. $\overline{y} = \frac{1}{n}\sum_{i=1}^{n} y_i$
  3. The error terms $\epsilon_i$ are uncorrelated with constant variance $\sigma^2$
  4. $y_i$, $\hat{\beta}_0$ and $\hat{\beta}_1$ are random variables.
  5. $x_i$ are known constants. $\beta_0$ and $\beta_1$ are the true values of the estimators, hence are also constants.

During the derivation we are going to use the following properties of variance. Here $a$ and $b$ are constants and $X$ and $Y$ are random variables.

  1. $\mathrm{Var}(aX+b) = a^2\mathrm{Var}(X)$
  2. $\mathrm{Var}(aX+bY) = a^2\mathrm{Var}(X) + b^2\mathrm{Var}(y) + 2ab \mathrm{Cov}\left(X,\:Y\right)$

We start with the definition of $\hat{\beta}_0$ : $$ \hat{\beta}_0 = \overline{y} - \hat{\beta}_1 \overline{x} $$

Therefore we have: $$ \mathrm{Var}\left(\overline{y} - \hat{\beta}_1 \overline{x}\right) $$

At this point if $\overline{y}$ and $\hat{\beta}_1$ are uncorrelated we can take each term's variance. This uncorrelation is what @Flowsnake's answer refers to. $$ \mathrm{Var}\left(\overline{y}\right) + \mathrm{Var}\left(-\hat{\beta}_1 \overline{x}\right) $$

As the predictors (inputs) $x_i$ are generally assumed to be known we can treat $\overline{x}$ as a constant. $$ \mathrm{Var}\left(\overline{y}\right) + \left(-\overline{x}\right)^2 \mathrm{Var}\left(\hat{\beta}_1\right) $$

Using @ThorGirl's answer, we can substitute $\mathrm{Var}\left(\hat{\beta}_1\right)$: $$ \mathrm{Var}\left(\overline{y}\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

The following steps will focus on $\mathrm{Var}\left(\overline{y}\right)$. As $y_i$ is not a constant, we must make a few substitutions. $$ \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^ny_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

By definition of the statistical model, we can further expand the term. $$ \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n\left(\beta_0 + \beta_1x_i + \epsilon_i\right)\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\mathrm{Var}\left(\sum_{i=1}^n\left(\beta_0 + \beta_1x_i + \epsilon_i\right)\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$

As $\epsilon_i$ is uncorrelated, the variance of the sum of random variables becomes a sum of individual variances. $$ \frac{1}{n^2}\sum_{i=1}^n\mathrm{Var}\left(\beta_0 + \beta_1x_i + \epsilon_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\sum_{i=1}^n\mathrm{Var}\left(\epsilon_i\right) + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n^2}\sum_{i=1}^n\sigma^2 + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \frac{1}{n}\sigma^2 + \overline{x}^2 \frac{\sigma^2}{\sum_{i=1}^n \left(x_i-\overline{x}\right)^2} $$ $$ \sigma^2 \left[\frac{1}{n} + \frac{\overline{x}^2}{\sum_{i=1}^n\left(x_i - \overline{x}\right)^2}\right] $$

$\endgroup$
3
  • 1
    $\begingroup$ You missed minus in $Var(a + b) = Var(a) - Var(b)$ in assuming uncorrelation $\endgroup$
    – GreatDuke
    Apr 11, 2020 at 19:39
  • $\begingroup$ @GreatDuke Can you give more detail ? Where is the error ? $\endgroup$
    – Hakan Baba
    Apr 11, 2020 at 19:47
  • $\begingroup$ The error is between the 2nd and 3rd line: Var(y − β x) = Var(y) + x^2 Var(β). $\endgroup$
    – CElliott
    Aug 23, 2021 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.