9
$\begingroup$

How might one understand the standard error (SE) of regression slope: $$s(b_1) = \sqrt{\frac{1}{n-2}·\frac{\sum{(y_i-\hat{y}_i)^2}}{\sum{(x_i-\bar{x})^2}}}$$

$\endgroup$
6
  • $\begingroup$ What do you mean by "understand"? What is it that you do not understand? As it stands the question is rather broad. $\endgroup$ Commented Apr 25, 2018 at 10:36
  • 2
    $\begingroup$ The numerator is residuals standard deviation which helps to get prediction interval this part is fine. The standard error is used for confidence interval for hypothesis testing. In linear regression i can reject null hypothesis i.e slope = 0 based on this standard error. My understanding of standard error is the standard deviation of the population parameter estimates produced by sampling distribution. Now the question is the formula can be seen in the same way or differently. I am looking for an intuitive understanding. $\endgroup$ Commented Apr 25, 2018 at 11:19
  • 2
    $\begingroup$ Yes, the question is about the actual elements comprising the formula $\endgroup$ Commented Apr 25, 2018 at 13:31
  • $\begingroup$ I looked a bit more carefully at your equation...this does not appear correct (as there is no covariance or correlation term) $\endgroup$
    – Gregg H
    Commented Apr 25, 2018 at 17:56
  • $\begingroup$ Scratch last comment...I miss read the hat as a bar...will edit the original question to make equation clear. $\endgroup$
    – Gregg H
    Commented Apr 25, 2018 at 19:33

1 Answer 1

8
$\begingroup$

The intuitive understanding is indeed as you suggest in the comment. If you think about the value of the slope for the regression as something that will change every time you draw a new sample (which it does), then the standard deviation of the resulting sampling distribution is the standard error for that parameter. So, if you imagine collecting 1,000 different samples (from the same population), then calculate the slope parameter for those different samples, you will have 1,000 slope estimates, and the standard deviation of those values will be very close to the calculated standard error produced by this formula.

Now, if the question is about the actual elements comprising the formula, this is much more challenging to explain "intuitively". I'll hold off from attempting that just yet, as the first part may have answered your question.


Update #1
The variability of the estimate for the slope parameter can be associated with the spread of the points about the population regression line. The more spread out the variability of these points about the line, the more "wiggle" you will see in the estimates that you might obtain. (I'd love to generate a graphic for this, alas...no time.)

So, to interpret the parts of the equation for the standard error for this parameter, we should rewrite the formula from the original post $$s(b_1) = \sqrt{\frac{1}{n-2}·\frac{\sum{(y_i-\hat{y}_i)^2}}{\sum{(x_i-\bar{x})^2}}}$$ as $$\begin{align}s(b_1) & = \sqrt{\frac{1}{n-2}·\frac{\sum{(y_i-\bar{y})^2}}{\sum{(x_i-\bar{x})^2}}·\frac{\sum{(y_i-\hat{y}_i)^2}}{\sum{(y_i-\bar{y})^2}}} \\ & = \sqrt{\frac{1}{n-2}}\sqrt{\frac{\frac{1}{n-1}\sum{(y_i-\bar{y})^2}}{\frac{1}{n-1}\sum{(x_i-\bar{x})^2}}}\sqrt{\frac{SS_\text{error}}{SS_\text{total}}} \\ & = \sqrt{\frac{1}{n-2}}\sqrt{\frac{sd(y)^2}{sd(x)^2}}\sqrt{\frac{SS_\text{total}-SS_\text{model}}{SS_\text{total}}} \\ & = \sqrt{\frac{1}{n-2}}\frac{sd(y)}{sd(x)}\sqrt{1-r^2} \end{align}$$ If were are willing to ignore the initial sample-size related scaling factor as a bias-adjustment, the next factor in the expression is a scaling factor along each of the dimensions. Imagine all the possible diagonal lines that might fit into a box. If you change the height or width of the box, it will change the amount of variability you might observe. The last factor is the measure of the spread of points about the line. The more spread, the more variability in the possible diagonal lines that might be observed (and thus, more variability in the slope). The less spread of points about the line, the less variability in the slope, and the less variability in the regression parameter.

$\endgroup$
7
  • 1
    $\begingroup$ The SE computed using the formula is based on zero as mean(thinking there is no slope)?, but as per your answer the means of 1000 slope estimates is actual mean or actual slope for the population. Both SEs will have same value? $\endgroup$ Commented Apr 25, 2018 at 13:28
  • 1
    $\begingroup$ Yes, you are correct, the hypothesis test is built under the assumption that $\beta_1=0$. Though I can't confirm this just now, I do believe the SE estimate is for the actual population slope, whether it is zero or not. (Another commenter may confirm before I can return to this.) $\endgroup$
    – Gregg H
    Commented Apr 25, 2018 at 13:30
  • 1
    $\begingroup$ Yes, the question is about the actual elements comprising the formula :) $\endgroup$ Commented Apr 25, 2018 at 17:06
  • $\begingroup$ I've updated my answer to provide clarification about the formula. $\endgroup$
    – Gregg H
    Commented Apr 25, 2018 at 21:07
  • 1
    $\begingroup$ @PandianLe $\hat{y_i}$ is the predicted value of $y_i$ (here using least squares regression coefficients), whereas $\bar{y}$ is the sample mean value of $y$. $\endgroup$ Commented Mar 1, 2021 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.