In the book Introduction to Statistical Learning, the authors describe the relation between predictor $X$ and response $Y$, by linear regression as: $$ Y = \beta_{0} + \beta_{1}X+\epsilon$$
Here, $\beta_0$ is the intercept term and $\beta_1$ is the slope. $\epsilon$ is the error term.
By minimizing the least squares criterion, the values of $\hat{\beta_0}$ and $\hat{\beta_1}$ are found to be: $$\hat\beta_1 = \frac {\sum_{i=1}^{n}{(x_i-\bar{x})(y_i-\bar{y})}}{\sum_{i=1}^{n}(x_i-\bar{x})^2}$$

$$\hat{\beta_0}=\bar{y}-\hat{\beta_1}\bar{x}$$ Finding this wasn't very hard. Next, the authors find the standard error(SE) of these parameters. They do so by firstly providing the following : $$Var(\hat\mu)=SE(\hat\mu)^2=\frac{\sigma^2}{n}$$ That is, $SE = \frac {\sigma}{\sqrt{n}}$ (where $\sigma$ is the standard deviation of each of the realizations $y_i$ of $Y$).
Next, the authors give the standard errors of both the parameters: $$SE(\hat\beta_0)^2=\sigma^2 \Big[ \frac{1}{n}+\frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\Big]$$
$$SE(\hat\beta_1)^2=\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}$$ Where $\sigma^2=Var(\epsilon)$. There seems no connection between the formulas found for the parameters, and their standard errors. In order to find the standard error, we must have the variance of both the parameters. But how can we find the variance of formulas? They're bound to give an exact value for a particular input.

  • 1
    $\begingroup$ Where did the parameter $\mu$ come from? $\endgroup$ – Michael R. Chernick Jul 8 '17 at 3:00
  • 1
    $\begingroup$ (According to the book) $\mu$ is the population mean of a random variable $Y$ $\endgroup$ – Mooncrater Jul 8 '17 at 3:01
  • $\begingroup$ Your statement "In order to find the standard error, we must have the standard deviation of both the parameters" suggests a possible misunderstanding on your part, or perhaps two: 1. The parameters are fixed (but unknown) quantities and have no standard deviation. 2. The standard deviation of the distribution of the parameter estimates is called the standard error of the parameters. We can estimate that standard error from the regression. $\endgroup$ – Glen_b -Reinstate Monica Jul 8 '17 at 11:28
  • $\begingroup$ @Glen_b Actually, by that line I wanted to put an emphasis on the fact that $$SE(\hat\mu)^2=var(\hat\mu)$$, which relates standard deviation and standard error. Meanwhile, I did have that doubt #1 in my head earlier. $\endgroup$ – Mooncrater Jul 8 '17 at 15:51
  • $\begingroup$ That actually relates the standard error of the mean to the variance of the sampling distribution of the estimate of the mean. $\endgroup$ – Glen_b -Reinstate Monica Jul 8 '17 at 15:54

Note $Var(\hat{\beta}_0) = Var(\bar{y} - \hat{\beta}_1\bar{x}) = Var(\bar{y}) + \bar{x}^2Var(\hat{\beta}_1) - 2Cov(\bar{y},\hat{\beta}_1)$. Try to show that the covariance term is 0.

The $Var(\hat{\mu}) = \dfrac{\sigma^2}{n}$ fact (although I'm not a fan of the notation they used here) is used in the calculation, $Var(\bar{y}) = \dfrac{\sigma^2}{n}$.

  • $\begingroup$ Due to my lack of experience and knowledge, I am unable to understand, how $cov(\bar{y},\hat{\beta_1})$ is $0$ (as, this is my first encounter with covariance). And after that, we still would need another relation between $var(\hat{\beta_0})$ and $var(\hat{\beta_1})$. That too, is too shy to be found by me. Sorry for my ignorance. $\endgroup$ – Mooncrater Jul 8 '17 at 10:05
  • 1
    $\begingroup$ @Mooncrater why is $Cov(\bar{y}, \hat{\beta}_{1}) = 0$? Hint: recall $\bar{y}$ is a constant. The formula for covariance of two r.v. $X$ and $Y$ is $Cov(X,Y) = E[XY] - E[X]E[Y]$. $\endgroup$ – JuliusBilly Jun 22 '18 at 3:21
  • $\begingroup$ Could you also derive $Var(\hat{\beta}_1)$ $\endgroup$ – Sabbiu Shah Jun 26 '19 at 1:27

Proof for standard error of parameters: $$SE(\hat{\beta_1})^2 =var\left(\frac{\sum_i(x_i-\bar{x})(y_i - \bar{y})}{\sum_i(x_i - \bar{x})^2}\right)$$ $Since \sum_i(x_i-\bar{x})(\bar{y}) = 0 =>$ $SE(\hat{\beta_1})^2 =var\left(\frac{\sum_i(x_i-\bar{x})(y_i)}{\sum_i(x_i - \bar{x})^2}\right)$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)var(\sum_i(x_i-\bar{x})y_i)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sum_i(x_i-\bar{x})^2var(y_i)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sum_i(x_i-\bar{x})^2(\sigma^2)$$ $$SE(\hat{\beta_1})^2 =\left(\frac{1}{(\sum_i(x_i - \bar{x})^2)^2}\right)\sigma^2\sum_i(x_i-\bar{x})^2$$ $$SE(\hat{\beta_1})^2 =\left(\frac{\sigma^2}{\sum_i(x_i - \bar{x})^2}\right)$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.