4
$\begingroup$

I'm new to statistics in general (but a very seasoned developer). I'm trying to grasp why it seems like there's a lot of consideration given to intercepts, at least where it comes to models.

For example, the formulas below were taken from here:

Fuel ~ 0 + Weight + Displacement      # formula with no intercept term
Fuel ~ -1 + Weight + Displacement     # formula with no intercept term

Whether you omit the intercept terms or set them to 0, -1, etc.., it's just a superficial adjustment, right? In the end, does the intercept have no more importance than to just represent a vertical translation of the model?

$\endgroup$
  • 3
    $\begingroup$ The "0" and "-1" don't set the intercept to those two values. The two are different ways of saying "omit the intercept effect". Well, arguably the first one is saying "set the intercept to 0", but the second one isn't saying "set it to -1". It's saying "remove the column of 1's from the model matrix", consistent with the use of "-" in formulas more generally in R. $\endgroup$ – Glen_b -Reinstate Monica Apr 13 '15 at 2:59
4
$\begingroup$

Representing vertical translation is important. In a regression setting, the intercept is the expected value of the response variable when all features are 0. Omitting the intercept when the intercept is not truly 0 also has the effect of causing incorrect estimation of the remaining regression. I constructed the following example to illustrate.

set.seed(31415)
x   <- rnorm(100)
y   <- 5+x+rnorm(100, sd=0.1)

plot(x,y, ylim=c(-2, 8))
model1  <- lm(y~x+1)
model2  <- lm(y~x+0)
abline(model1)
abline(model2)

In this case, we know the true model is $y=5+x+\epsilon$, but if we estimate the model without the intercept, the regression line is nowhere near the data!

enter image description here

We can also see that the estimates of the coefficient of $x$ are incorrect. The correctly-specified model estimates $x$ very close to 1, while the model omitting the intercept is not at all close to 1. It's hard to discern the effect in the graph, though.

model1

Call:
lm(formula = y ~ x + 1)

Coefficients:
(Intercept)            x  
      4.991        1.000

model2

Call:
lm(formula = y ~ x + 0)

Coefficients:
     x  
1.514

You're correct that in most cases you won't know the value of the intercept ahead of time. In general, the problems associated with omitting a feature is called "omitted variable bias." If the true model had two features $x_1$ and $x_2$, but we omitted one of them in the model, we would similarly have incorrect estimates of the remaining parameters.

You can look at the confidence interval of the estimate to get a sense of how strongly the data support the hypothesis of a nonzero intercept.

If all variables are transformed by subtracting their mean values, this effectively eliminates the need to estimate an intercept term, because the expectation of $y$ must be $0$ when all the features are also $0$. This comes at the cost of interpretability, though, since you'll be working on a new "scale." For example, if $x$ is temperature in Celsius, the $0$-mean temperature variable $x^\prime$ has moved the zero point to be the mean value of the data, so all of the nice properties about the location of the freezing point of water and so on move along with it. Whether this loss in direct, obvious interpretation is important to you depends on the particular application, of course.

$\endgroup$
  • 2
    $\begingroup$ You really put it in perspective for me. I hadn't considered that: "Omitting the intercept when the intercept is not truly 0 also has the effect of causing incorrect estimation of the remaining regression". It critically skews everything. $\endgroup$ – Dustin Oprea Apr 13 '15 at 1:30
  • $\begingroup$ How would you know the value ahead of time? Shouldn't this be part of the model? Doesn't figuring out the intercept fall in line with the guessing of the coefficients that it's already doing? $\endgroup$ – Dustin Oprea Apr 13 '15 at 1:32
  • 2
    $\begingroup$ +1, another way to frame this is that you cannot omit the intercept, only force the intercept to be $0$. (On a different note, it is often helpful to set the seed w/ R demonstrations so that people can reproduce them exactly.) $\endgroup$ – gung - Reinstate Monica Apr 13 '15 at 1:54
  • 2
    $\begingroup$ @DustinOprea Oh! In R, you can explicitly specify the inclusion of an intercept in the formula interface with +1. Excluding the intercept is done with -1 or +0; intercepts are also included by default, I just wanted to make the code explicit. In terms of syntax, it's a binary indicator, I believe. More detail can be found in ?formula. $\endgroup$ – Reinstate Monica Apr 13 '15 at 2:30
  • 1
    $\begingroup$ @user777 Ok. We're on the same page :) . The help is too terse to be helpful at this stage in the game for me. $\endgroup$ – Dustin Oprea Apr 13 '15 at 3:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.