1
$\begingroup$

In the book " Introduction to Statistical Learning " , the standard error of the slope term of Linear Regression is given as follows :

enter image description here

The book also says the Slope is more precise when the data is more spread out ( As suggested by the term in the denominator ) , but I am not able to visualize this .

Can someone please provide a visual reasoning of why the slope is more precise and its standard error less when the input data X is more spread out ?

$\endgroup$
4
  • 2
    $\begingroup$ balance a long stick on two fingers. When the fingers are far apart, wiggling them up and down a bit (effect of a little noise) changes the slope of the stick only a little. if you move the fingers closer together, the same amount of movement changes the slope much more. $\endgroup$
    – Glen_b
    Commented Jul 9, 2020 at 11:20
  • $\begingroup$ I got it ! Thanks $\endgroup$
    – Bharathi
    Commented Jul 9, 2020 at 14:27
  • $\begingroup$ @Bharathi my inbox mentioned you left a comment on my answer, but I don't see it? was that a mistake or did something mess up ? $\endgroup$
    – doubled
    Commented Jul 9, 2020 at 15:55
  • 1
    $\begingroup$ @doubled , Yes , I did comment asking you for a graph . But deleted since I understood it. :) $\endgroup$
    – Bharathi
    Commented Jul 9, 2020 at 17:45

1 Answer 1

2
$\begingroup$

Let me know if it helps to have graphs of this, but I think the following should be enough to 'visualize' this observation:

Suppose that the variance of $X$ is $\sigma^2 = 1$. Then your equation says that $$SE(\hat{\beta})^2 = \frac{1}{\sum_{i=1}^n(x_i - \bar{x})^2}$$

Just studying the function $f(t) = 1/t$ for $t$ positive, we can see that if $t < t'$, then $1/t > 1/t'$. So in words, larger values of the denominator means the function value is smaller, and relating that to your formula, we see that larger values of the denominator means that the standard error is smaller.

For each $x_i$, $(x_i-\bar{x})^2$ is a measure of how far away $x_i$ is from the sample mean. Consider two cases. In the first case, the $x_i$'s are all very close to each other, and thus very close to the mean of these values. Then each $x_i - \bar{x}$ is quite small, and thus so is $(x_i - \bar{x})^2$, so the sum of all of them is also relatively small. Small denominator means std error is large.

In the second case, the $x_i$'s are all very far apart from each other, suppose we did this by taking the previous $x_i$'s and either subtract or add some amount to all of them essentially pushing them away from $\bar{x}$ while keeping $\bar{x}$ the same as in the previous case. Now, each $x_i - \bar{x}$ is the original difference plus the additional amount we pushed them away from the mean, but when you sum them together for the sample mean, the increased negative and positive values 'cancel out', so the mean stays unchanged. However, in $\sum_{i=1}^n (x_i-\bar{x})^2$, we are taking each difference and squaring them, so each value adds a positive amount to the total and there is no canceling out. So now each $(x_i-\bar{x})^2$ is now larger than in the first case, and that amount adds to the size of the denominator. Larger denominator means std error is smaller.

Essentially, if you have two cases with the same sample mean, but one has more spread out data, you're gonna get larger $(x_i-\bar{x})^2$ for the spread out case, and so the sum is going to be larger, and larger denominator (fixing numerator $\sigma^2$) will make the expression smaller, thus smaller std error.

In case it helps illustrate, suppose case one described above has $x_i$'s, and suppose for one value $x_1$ below the mean, we instead observed $x_1' = x_1 - 1000$, and for one value $x_2$ above the mean, we instead observed $x_2' = x_2 + 1000$. Then the sample mean with these observations, $\bar{x}'$ is $$\bar{x}' = \frac{1}{n}\sum_{i=1}^n x_i' = \frac{1}{n}\bigg(x_1' + x_2' + \sum_{i=3}^n x_i\bigg) = \frac{1}{n}\bigg(x_1 - 1000 + x_2 + 1000 + \sum_{i=3}^n x_i\bigg) = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x}$$

so the mean did not change, but looking at $\sum_{i=1}^n (x_i - \bar{x}^2)$, for $i=1,2$, we now have $$(x_1' - \bar{x})^2 = (x_1 - \bar{x} - 1000)^2 $$ but $x_1$ was already below the mean, so $x_1 - \bar{x}$ is a negative number, and then we are further taking off another $1000$ and then squaring, so this number is strictly bigger than before! And same argument for $x_2'$. So the denominator got strictly a lot bigger!

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.