1
$\begingroup$

I'm a bit confused regarding how intercept restrictions impact slope estimators in a simple linear regression. An example:

$$H_0:\beta_0=0$$ $$H_A:\beta_0\neq 0$$

I simulated a variable $y_i$ as:

$$y_i= \beta_0 + \beta x_i +\epsilon_i \quad \cdots(1)\quad\quad \beta_0\neq0$$

$$y_i= 0 + \beta x_i +\epsilon_i \quad \quad \cdots(2)$$

fitting a simple linear regression on each of the simulations yields the exact same estimators for $\beta$ under both least squares and maximum likelihood.

Now if I fit simple linear regressions while including the restriction on the intercept I get two different estimators for each of the possible simulated variables. Can someone help me understand what's going on?

What confuses me the most is the fact that the restricted and unrestricted slope estimators are not the same even when the true model for $y_i$ has no intercept.

I'm attaching my R code for reference:

x<-rnorm(100) 
err<-5*rnorm(100)
y1 <- 5*x+err
y2 <- 5+5*x+err
summary(lm(y1~x))
summary(lm(y2~x))
summary(lm(y1~x+0))
summary(lm(y2~x+0))
Improve this question
$\endgroup$
4
  • $\begingroup$ Hi: In order to run lm in R without intercept, you need to use -1 rather than zero. The zero isn't doing anything. Also, MLE and least squares are identical for normal error term assumption. $\endgroup$
    – mlofton
    Commented Jul 30, 2020 at 22:18
  • 1
    $\begingroup$ I'm pretty sure the "-1" and "+0" are just different code formats that yield the same results. $\endgroup$
    – tvbc
    Commented Jul 30, 2020 at 22:21
  • 1
    $\begingroup$ @mlofton In R model formulas +0 suppresses the intercept, as does -1. I think the first makes more sense in regression applications, while the second makes more sense in anova. Both work the same; just try it: lm(dist~speed-1,cars);lm(dist~speed+0,cars). Also see ?formula which says (in discussing the - operator in the formula interface): It can also used to remove the intercept term: when fitting a linear model y ~ x - 1 specifies a line through the origin. A model with no intercept can be also specified as y ~ x + 0 or y ~ 0 + x. $\endgroup$
    – Glen_b
    Commented Jul 31, 2020 at 4:03
  • $\begingroup$ interesting and thanks for info, that behavior must have changed at some point because it used to be only -1.0. ( in the dark ages. i've been using R for a while ). thanks. $\endgroup$
    – mlofton
    Commented Jul 31, 2020 at 23:30

2 Answers 2

Reset to default
2
$\begingroup$

I think the slope in a regression with unrestricted intercept is not influenced by the intercept. It's determined solely by the deviance of x and y over their means. As a consequence, the information of position of y (the intercept) is not influencing the slope, which is calculated in a second step.

This can be seen in the formula for the least squares estimation of the slope:

$$\hat{\beta} = \frac{\sum_i(x_i-\bar{x})(y_i-\bar{y})}{\sum_i(x_i-\bar{x})^2}$$

As you can see, all y values are subtracted from its mean, which means that this value would not change if you add +5 to all y points.

On the other hand, a restricted regression has no concept of intercept. It calculates the slope as:

$$ \hat{\beta} = \frac{\sum_i x_iy_i}{\sum_i x_i^2} $$

This assumes that the intercept passes through 0. If it does not, the intercept is subtracted from all y. In that case subtracting the 5 will influence the values of y. So results will differ especially if the sample has a different optimal intercept than the one you restricted data to.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That makes sense but I still can’t see why fitting the simple linear model under the 0-intercept simulated variable yields different slopes for the restricted and unrestricted models. My guess so far is that the unrestricted model does not “know” the intercept is zero so it fits the closest thing to the data which should, most of the time, produce an intercept that’s close to but not necessarily zero. $\endgroup$
    – tvbc
    Commented Jul 30, 2020 at 23:39
  • $\begingroup$ In that case both slope estimators would eventually converge. This leads me to wonder about a second thing though, when doing a likelihood ratio test for the hypothesis, which $\hat \beta_1$ should I use? $\endgroup$
    – tvbc
    Commented Jul 30, 2020 at 23:42
0
$\begingroup$

When you tell lm not to include an intercept, you force the regression line to go through your cloud of points, probably somewhere near $(\bar{x},\bar{y})$, and the origin. The odds are low that the slope of such a line is the same as what you would fit to just the cloud of points.

If the $y$-intercept was positive in the unconstrained linear model, forcing it to 0 will require the slope coefficient to be more positive. If the $y$-intercept was negative, forcing it to 0 will yield a more negative slope.

In general, there are almost never good reasons to suppress the intercept. Even if you expect the intercept to be 0, allowing for a non-zero value lets you correct for imbalances or trends in the data -- which you might not expect to continue in the future.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ note that, in addition to what kurtosis said, all the rsquared machinery goes out the window ( i.e. it's not applicable ) once you don't include the intercept in your model. $\endgroup$
    – mlofton
    Commented Jul 31, 2020 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.