8
$\begingroup$

In the book Introduction to Statistical Learning, the authors describe the relation between predictor $X$ and response $Y$, by linear regression as: $$ Y = \beta_{0} + \beta_{1}X+\epsilon$$
Here, $\beta_0$ is the intercept term and $\beta_1$ is the slope. $\epsilon$ is the error term.
By minimizing the least squares criterion, the values of $\hat{\beta_0}$ and $\hat{\beta_1}$ are found to be: $$\hat\beta_1 = \frac {\sum_{i=1}^{n}{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^{n}(x_i-\bar{x})^2}$$

$$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ Finding this wasn't very hard. Next, the authors find the standard error(SE) of these parameters. They do so by firstly providing the following : $$Var(\hat\mu)=SE(\hat\mu)^2=\frac{\sigma^2}{n}$$ That is, $SE = \frac {\sigma}{\sqrt{n}}$ (where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$).
Next, the authors give the standard errors of both the parameters: $$SE(\hat\beta_0)^2=\sigma^2 \Big[ \frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\Big]$$
$$SE(\hat\beta_1)^2=\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}$$ Where $\sigma^2=Var(\epsilon)$. There seems no connection between the formulas found for the parameters, and their standard errors. In order to find the standard error, we must have the variance of both the parameters. But how can we find the variance of formulas? They're bound to give an exact value for a particular input.

$\endgroup$
  • 1
    $\begingroup$ Where did the parameter $\mu$ come from? $\endgroup$ – Michael R. Chernick Jul 8 '17 at 3:00
  • 1
    $\begingroup$ (According to the book) $\mu$ is the population mean of a random variable $Y$ $\endgroup$ – Mooncrater Jul 8 '17 at 3:01
  • 1
    $\begingroup$ Your statement "In order to find the standard error, we must have the standard deviation of both the parameters" suggests a possible misunderstanding on your part, or perhaps two: 1. The parameters are fixed (but unknown) quantities and have no standard deviation. 2. The standard deviation of the distribution of the parameter estimates is called the standard error of the parameters. We can estimate that standard error from the regression. $\endgroup$ – Glen_b Jul 8 '17 at 11:28
  • $\begingroup$ @Glen_b Actually, by that line I wanted to put an emphasis on the fact that $$SE(\hat\mu)^2=var(\hat\mu)$$, which relates standard deviation and standard error. Meanwhile, I did have that doubt #1 in my head earlier. $\endgroup$ – Mooncrater Jul 8 '17 at 15:51
  • $\begingroup$ That actually relates the standard error of the mean to the variance of the sampling distribution of the estimate of the mean. $\endgroup$ – Glen_b Jul 8 '17 at 15:54
3
$\begingroup$

Note $Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2Var(\hat{\beta}_1) - 2Cov(\bar{y},\hat{\beta}_1)$. Try to show that the covariance term is 0.

The $Var(\hat{\mu}) = \dfrac{\sigma^2}{n}$ fact (although I'm not a fan of the notation they used here) is used in the calculation, $Var(\bar{y}) = \dfrac{\sigma^2}{n}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Due to my lack of experience and knowledge, I am unable to understand, how $cov(\bar{y},\hat{\beta_1})$ is $0$ (as, this is my first encounter with covariance). And after that, we still would need another relation between $var(\hat{\beta_0})$ and $var(\hat{\beta_1})$. That too, is too shy to be found by me. Sorry for my ignorance. $\endgroup$ – Mooncrater Jul 8 '17 at 10:05
  • 1
    $\begingroup$ @Mooncrater why is $Cov(\bar{y}, \hat{\beta}_{1}) = 0$? Hint: recall $\bar{y}$ is a constant. The formula for covariance of two r.v. $X$ and $Y$ is $Cov(X,Y) = E[XY] - E[X]E[Y]$. $\endgroup$ – JuliusBilly Jun 22 '18 at 3:21
  • $\begingroup$ Could you also derive $Var(\hat{\beta}_1)$ $\endgroup$ – Sabbiu Shah Jun 26 '19 at 1:27
3
$\begingroup$

Proof for standard error of parameters: $$SE(\hat{\beta_1})^2 =var\left(\frac{\sum_i(x_i-\bar{x})(y_i - \bar{y})}{\sum_i(x_i - \bar{x})^2}\right)$$ $Since \sum_i(x_i-\bar{x})(\bar{y}) = 0 =>$ $SE(\hat{\beta_1})^2 =var\left(\frac{\sum_i(x_i-\bar{x})(y_i)}{\sum_i(x_i - \bar{x})^2}\right)$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)var(\sum_i(x_i-\bar{x})y_i)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sum_i(x_i-\bar{x})^2var(y_i)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sum_i(x_i-\bar{x})^2(\sigma^2)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sigma^2\sum_i(x_i-\bar{x})^2$$ $$SE(\hat{\beta_1})^2 =\left(\frac{\sigma^2}{\sum_i(x_i - \bar{x})^2}\right)$$

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ If anyone are looking for a step-by-step explanation of @ThorGirl's answer take a look at this video. $\endgroup$ – Hakan Baba Mar 11 at 7:23
0
$\begingroup$

Once we have @ThorGirl's derivation for $ SE(\hat{\beta_1})^2 $ we can use that to derive the Standard Error for $ \hat{\beta_0} $ i.e, $ SE(\hat{\beta_0})^2 $

Note: If you are looking for a step-by-step explanation of @ThorGirl's answer take a look at this video.

We are going to use the following assumptions / observations:

1) Each output $y_i$ is assumed to have this form: $ \beta_0 + \beta_1x_i + \epsilon_i$

2) $\overline{y} = \frac{1}{n}\sum_{i=1}^{n} y_i$

2) The error terms $ \epsilon_i $ are uncorrelated with constant variance $ \sigma^2$

3) The $y_i$'s, $\hat{\beta}_0$ and $\hat{\beta}_1$ are random variables.

4) The $x_i$'s are known constants. The $\beta_0$ and $\beta_1$ are not necessarily known but they are also constants, they are not random variables.

During the derivation we are going to use the following properties of variance. Here $a$ is a constant and $x$ and $y$ are random variables.

1) $Var(a+x) = Var(x) $

2) $Var(ax) = a^2Var(x) $

3) $ Var(x+y) = Var(x) + Var(y) + 2 Cov(a,b) $

We start with the definition of $\hat{\beta}_0$ :

$$ \overline{y} - \hat{\beta}_1 \overline{x } = \hat{\beta}_0 $$

Take the variance of it to find the standard error:

$$ Var( \overline{y} - \hat{\beta}_1 \overline{x }) $$

At this point if $\overline{y}$ and $\hat{\beta}_1$ are uncorrelated we can take each term's variance. This uncorrelation is what @Flowsnake's answer refers to. That uncorrelation is not being proven here.

$$ Var( \overline{y}) - Var(\hat{\beta}_1 \overline{x }) $$

The $\overline{x}$ is constant here. It is generally assumed that the predictors, inputs $x$'es are known. Take that out of the variance.

$$ Var( \overline{y}) - \overline{x}^2 Var(\hat{\beta}_1) $$

@ThorGirl's answer has the value for $Var(\hat{\beta}_1)$ substitute that:

$$ Var( \overline{y}) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

Now going to focus on the $Var(\overline{y})$. That is not a constant. Write its open form:

$$ Var( \frac{1}{n} \sum_{i=1}^{n}y_i) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

We also know the $y_i$ terms by the definition of the model substitute that too.

$$ Var( \frac{1}{n} \sum_{i=1}^{n} (\beta_0 + \beta_1x_i + \epsilon_i)) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

Take the $1/n$ out of the variance.

$$ \frac{1}{n^2} Var( \sum_{i=1}^{n} (\beta_0 + \beta_1x_i + \epsilon_i)) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

The $\epsilon_i$ 's are uncorrelated random variables. With that, the variance of sum of random variables becomes the sum of their individual variances.

$$ \frac{1}{n^2} \sum_{i=1}^{n} Var(\beta_0 + \beta_1x_i + \epsilon_i) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

The constants ($\beta_0 + \beta_1x_i$) in the variance term can be ignored. $Var(a + x) = Var(x) $

$$ \frac{1}{n^2} \sum_{i=1}^{n} Var(\epsilon_i) - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

By our assumptions, the variance of each $\epsilon_i$ random variable is constant and is $\sigma^2$ . Replace that.

$$ \frac{1}{n^2} \sum_{i=1}^{n} \sigma^2 - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

Summing $\sigma^2$ $n$ times. Simplify that and cancel $n$ from the nominator and denominator

$$ \frac{1}{n^2} n \sigma^2 - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

$$ \frac{1}{n} \sigma^2 - \overline{x}^2 \frac{\sigma^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } $$

Factor out the $\sigma^2 $

$$ \sigma^2 \left [ \frac{1}{n} - \frac{\overline{x}^2}{ \sum_{i=1}^ {n} (x_i-\overline{x})^2 } \right ] $$

We found the $SE(\hat{\beta_0})^2$

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You missed minus in $Var(a + b) = Var(a) - Var(b)$ in assuming uncorrelation $\endgroup$ – GreatDuke Apr 11 at 19:39
  • $\begingroup$ @GreatDuke Can you give more detail ? Where is the error ? $\endgroup$ – Hakan Baba Apr 11 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.