5
$\begingroup$

Let's say we have data that looks like this:

set.seed(1)
b0 <- 0 # intercept
b1 <- 1 # slope
x <- c(1:100) # predictor variable
y <- b0 + b1*x + rnorm(n = 100, mean = 0, sd = 200) # predicted variable

We fit a simple linear model:

mod.1 <- lm(y~x) 
summary(mod.1) 
#             Estimate   Std. Error  t value  Pr(>|t|)
# (Intercept) 26.3331    36.3795     0.724    0.471
# x           0.9098     0.6254      1.455    0.149 
b0.est <- summary(mod.1)$coefficients[1,1]
b1.est <- summary(mod.1)$coefficients[2,1]

And a model where we (1) subtract off the intercept term fit in the first model from the dataset and (2) prevent the intercept term from being fit (or in other words, force the model through zero):

mod.2  <- lm(y - b0.est  ~ 0 + x) 
summary(mod.2) 
#             Estimate   Std. Error t value   Pr(>|t|)   
# x           0.9098     0.3088     2.946     0.00401 **
b1.est.2 <- summary(mod.2)$coefficients[1,1]

As to be expected the slope parameter stays the same (0.9098).

However, while the slope parameter was not significant in the first model, it is in the second model (the standard error on the estimate in the second model is much lower than in the first model, 0.3088 vs. 0.6254).

The data is the same shape in both models with the same slope parameter being estimated by the two models. How is it the second model is so much more "certain" of the slope parameter estimate?

Or to put it another way, how are these standard errors calculated?

Using the equation for standard error I found here, I calculated the standard errors for model 1 and 2 this way:

# Model 1
DF <- length(x)-2 
y.est <- b0.est + b1.est*x 
numerator <- sqrt(sum((y - y.est)^2)/DF) 
denominator <- sqrt(sum((x - mean(x))^2))
numerator/denominator 
# SE = 0.6254

This matches the R output.

# Model 2
DF <- length(x)-1 
y.est <- b1.est.2*x 
numerator <- sqrt(sum((y - (y.est+b0.est))^2)/DF) 
denominator <- sqrt(sum((x - mean(x))^2))
numerator/denominator 
# SE = 0.6223

This doesn't match the R output which has the SE = 0.3088.

What am I missing?

$\endgroup$
  • $\begingroup$ I think there is something off with the call mod.2 <- lm(y - b0.est ~ 0 + x). I'm not sure that this is the way to eliminate the intercept. Compare the slope when you do the regression through the origin with lm(y ~ x - 1). $\endgroup$ – Antoni Parellada Dec 19 '15 at 4:22
  • $\begingroup$ @Antoni That approach looks fine to me. The procedure is illustrated in my answer. $\endgroup$ – whuber Dec 19 '15 at 4:52
8
$\begingroup$

The formulas are the same as always, so let's focus on understanding what's going on.

Here is a small cloud of points. Its slope is uncertain. (Indeed, the coordinates of these points were drawn independently from a standard Normal distribution and then moved a little to the side, as shown in subsequent plots.)

Figure 1

Here is the OLS fit. The intercept is near $3$. That's kind of an accident: the OLS line must pass through the center of mass of the point cloud and where the intercept is depends on how far I moved the point cloud away from the origin. Due to the uncertain slope and the relatively large distance the points were moved to the right, the intercept could be almost anywhere. To illustrate, the slopes of the dashed lines differ from the fitted line by up to $\pm 1/2$. All of them fit the data pretty well.

OLS

After lowering the cloud by the height of the intercept, the OLS line (solid gray) goes through the origin, as expected.

OLS fit to lowered data

The OLS line remains just as uncertain as it was before. The standard error of its slope is high. But if you were to constrain it to pass through the origin, the only wiggle room left is to vary the other end up and down through the point cloud. The dotted lines show the same range of slopes as before: but now the extreme ones don't go anywhere near the cloud. Constraining the fit has greatly increased the certainty in the slope.

$\endgroup$
  • $\begingroup$ I was trying to get some relation between the high SE of the intercept and the lower SE of the slope in the model with an intercept. My intuition was that somehow the uncertainty of the of the slope was absorbed by the intercept. And then came your answer... $\endgroup$ – Antoni Parellada Dec 19 '15 at 4:58
  • $\begingroup$ BTW. The concept is now clear, but is there a mathematical equation relating the decrease in SE of the slope to the inclusion or exclusion of an intercept? $\endgroup$ – Antoni Parellada Dec 19 '15 at 4:59
  • 2
    $\begingroup$ @Antoni Of course there must be. It's potentially complicated, because it's exactly the same as the equations describing how the SE of any estimated coefficient changes when other regressors are added to or removed from a model. The whole mathematical point is that the intercept is just another regressor in the model. $\endgroup$ – whuber Dec 19 '15 at 5:04
  • $\begingroup$ Thank you for such a wonderfully illustrated and intuitive answer. It helps me enormously. Many thanks again! $\endgroup$ – Angela Dec 20 '15 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.