2
$\begingroup$

The answer here explains, why the residuals of an OLS-regression have mean zero if an intercept is included.

Problem:

Intuitively, i would assume that including an intercept just "de-means" the residuals of the same regression without intercept. However, this seems not to be right:

set.seed(123)
x <- 1:100
e <- rnorm(100, sd = 10)
y <- x+e
# OLS-regression with intercept
summary(lm(y~x))

Call:
lm(formula = y ~ x)

Residuals:
     Min       1Q   Median       3Q      Max 
-24.5356  -5.5236  -0.3462   6.4850  20.9487 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.36404    1.84287  -0.198    0.844    
x            1.02511    0.03168  32.356   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 9.145 on 98 degrees of freedom
Multiple R-squared:  0.9144,    Adjusted R-squared:  0.9135 
F-statistic:  1047 on 1 and 98 DF,  p-value: < 2.2e-16
# OLS-regression without intercept
reg <- lm(y~x-1)
summary(reg)

Call:
lm(formula = y ~ x - 1)

Residuals:
     Min       1Q   Median       3Q      Max 
-24.5085  -5.6817  -0.3652   6.2934  20.8238 

Coefficients:
  Estimate Std. Error t value Pr(>|t|)    
x  1.01968    0.01565   65.17   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 9.101 on 99 degrees of freedom
Multiple R-squared:  0.9772,    Adjusted R-squared:  0.977 
F-statistic:  4247 on 1 and 99 DF,  p-value: < 2.2e-16
# mean of residuals
mean(reg$residuals)
[1] -0.08965128

We see, that the estimated intercept has a value of -0.36404 (and the residuals have mean zero). The same model without intercept reports a mean for the residuals of -0.08965.

Question:

The intercept does not just "de-mean" the residuals, so what is the relationship between the intercept of an OLS-regression and the residuals of the same model without intercept?

$\endgroup$
5
$\begingroup$

In simple linear regression with intercept, $y = \alpha + \beta x$, the least squares estimates are $$ \hat{\beta}_I = \frac{\sum_ix_iy_i - n\bar{x}\bar{y}}{\sum_ix_i^2 - n\bar{x}^2}, \qquad \hat{\alpha}_I = \bar{y} - \hat{\beta}_I\bar{x}. $$ In simple linear regression without intercept, $y = \beta x$, the least squares estimate is $$ \hat{\beta}_O = \frac{\sum_ix_iy_i}{\sum_ix_i^2}, $$ and so the residual mean is $$ \bar{r} = \frac{1}{n}\sum_iy_i - \hat{\beta}_Ox_i = \bar{y} - \hat{\beta}_O\bar{x}. $$ In the first model, $\hat{\alpha}_I = \bar{y} - \hat{\beta}_I\bar{x}$ while in the second model, $\bar{r} = \bar{y} - \hat{\beta}_O\bar{x}$. But the estimates for the $\beta$ coefficient are different unless the predictor(s) have mean 0. Then $\bar{x} = 0$ and the formula for $\hat{\beta}_I$ simplifies to the formula for $\hat{\beta}_O$.

Note: For clarity I've used the subscript $I$ to indicate the case with intercept and $O$ -- the case without.

And so you can combine the formulas for $\hat{\alpha}_I$ and $\bar{r}$ to derive the relationship: $$ \hat{\alpha}_I = \bar{r} + (\hat{\beta}_O- \hat{\beta}_I)\bar{x} $$

$\endgroup$
  • $\begingroup$ Thanks for the helpful explanation, but furthermore i would be interested in a more detailed description of the relationship between $\hat{\alpha}$ and $\bar{r}$, so does there exist a functional relation between those two parameters? $\endgroup$ – skoestlmeier Mar 5 at 9:43
  • $\begingroup$ Do you have in mind a particular case when there is a meaningful choice between including the intercept or not? What do you mean by a functional relationship? $\endgroup$ – dipetkov Mar 5 at 22:21
  • $\begingroup$ No, my question arises in general with no specific case in mind (i am also aware of this discussion about the intercept). With functional relationship i mean if there is defined a mapping between $\hat{\alpha}_I$ and $\bar{r}$, i.e. a function $f(\cdot)$ where $\hat{\alpha}_I=f(\bar{r})$ holds. $\endgroup$ – skoestlmeier Mar 8 at 9:51
  • 1
    $\begingroup$ Sure you can by adding and subtracting a $\hat{\beta}\bar{x}$ term appropriately. $\endgroup$ – dipetkov Mar 8 at 11:19
3
$\begingroup$

including an intercept just "de-means" the residuals of the same regression without intercept

You're suggesting that the two fits should be parallel.

Just look at a plot:

linear relationship with usual fit and fit through the origin

You're saying that the green line should be parallel to the red line. That doesn't make sense, you can get much closer to the data than that.

$\endgroup$
  • $\begingroup$ Sometimes, a simple picture says more than thousands words (+1)! Also as commented on the other answer, is there a way to calculate the intercept of the regression, given the residuals of the regression without intercept? $\endgroup$ – skoestlmeier Mar 8 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.