Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am running linear regression models and wondering what the conditions are for removing the intercept term of lm()?

In comparing results from two different lm functions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?

Any advice is appreciated.

share|improve this question
    
@chi thanks for editing my question. are there things that I should be clarifying or rewording in any future questions? –  analyticsPierce Mar 7 '11 at 15:00
    
Your question is well stated. @chl kindly improved some formatting, that's all. It involved TeXifying the "R^2" (it was turned into $\$$R^2$\$$, which renders as $R^2$). –  whuber Mar 7 '11 at 15:47
    
What would the intercept mean in your model? From the information in your question, it seems it would be the expected value of your response when sqft=0 and lotsize=0 and baths=0. Is that ever going to occur in reality? –  timbp Feb 27 '12 at 2:59
    
the intercept has no meaning. so when writing up a formula for the expected value of a house, can I leave this out? –  Travis Feb 27 '12 at 3:02
1  
NB: Some of these comments and replies address essentially the same question (framed in the context of a housing price regression) which was merged with this one as a duplicate. –  whuber Feb 27 '12 at 4:51
show 2 more comments

6 Answers

up vote 29 down vote accepted

The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult Brooks Introductory Econometrics for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.

For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is "much" higher.

Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).

share|improve this answer
    
Great summary, thank you for the insight. –  analyticsPierce Mar 7 '11 at 15:06
    
-1 for "never", see example 1 of Joshuas' answer –  Curious Dec 2 '13 at 20:43
add comment

Removing the intercept is a different model, but there are plenty of examples where it is legitimate. Answers so far have already discussed in detail the example where the true intercept is 0. I will focus on a few examples where we may be interested in an atypical model parametrization.

Example 1: The ANOVA-style Model. For categorical variables, we typically create binary vectors encoding group membership. The standard regression model is parametrized as intercept + k - 1 dummy vectors. The intercept codes the expected value for the "reference" group, or the omitted vector, and the remaining vectors test the difference between each group and the reference. But in some cases, it may be useful to have each groups' expected value.

dat <- mtcars
dat$vs <- factor(dat$vs)

## intercept model: vs coefficient becomes difference
lm(mpg ~ vs + hp, data = dat)

Coefficients:
(Intercept)          vs1           hp  
   26.96300      2.57622     -0.05453  

## no intercept: two vs coefficients, conditional expectations for both groups
lm(mpg ~ 0 + vs + hp, data = dat)

Coefficients:
     vs0       vs1        hp  
26.96300  29.53922  -0.05453  

Example 2: The case of standardized data. In some cases, one may be working with standardized data. In this case, the intercept is 0 by design. I think a classic example of this was old style structural equation models or factor, which operated just on the covariance matrices of data. In the case below, it is probably a good idea to estimate the intercept anyway, if only to drop the additional degree of freedom (which you really should have lost anyway because the mean was estimated), but there are a handful of situations where by construction, means may be 0 (e.g., certain experiments where participants assign ratings, but are constrained to give out equal positives and negatives).

dat <- as.data.frame(scale(mtcars))

## intercept is 0 by design
lm(mpg ~ hp + wt, data = dat)

Coefficients:
(Intercept)           hp           wt  
  3.813e-17   -3.615e-01   -6.296e-01  

## leaving the intercept out    
lm(mpg ~ 0 + hp + wt, data = dat)

Coefficients:
     hp       wt  
-0.3615  -0.6296  

Example 3: Multivariate Models and Hidden Intercepts. This example is similar to the first in many ways. In this case, the data has been stacked so that two different variables are now in one long vector. A second variable encodes information about whether the response vector, y, belongs to mpg or disp. In this case, to get the separate intercepts for each outcome, you suppress the overall intercept and include both dummy vectors for measure. This is a sort of multivariate analysis. It is not typically done using lm() because you have repeated measures and should probably allow for the nonindepence. However, there are some interesting cases where this is necessary. For example when trying to do a mediation analysis with random effects, to get the full variance covariance matrix, you need both models estimated simultaneously, which can be done by stacking the data and some clever use of dummy vectors.

## stack data for multivariate analysis
dat <- reshape(mtcars, varying = c(1, 3), v.names = "y",
  timevar = "measure", times = c("mpg", "disp"), direction = "long")
dat$measure <- factor(dat$measure)

## two regressions with intercepts only
lm(cbind(mpg, disp) ~ 1, data = mtcars)

Coefficients:
             mpg     disp  
(Intercept)   20.09  230.72

## using the stacked data, measure is difference between outcome means
lm(y ~ measure, data = dat)

Coefficients:
(Intercept)   measurempg  
      230.7       -210.6  

## separate 'intercept' for each outcome
lm(y ~ 0 + measure, data = dat)

Coefficients:
measuredisp   measurempg  
     230.72        20.09  

I am not arguing that intercepts should generally be removed, but it is good to be flexible.

share|improve this answer
3  
+1. I didn't take people to be rigidly saying 'never', but it's always nice to have another perspective & this is a very clear & thoughtful response. Welcome to CV, it'll be great to have you as part of the community. –  gung Jul 18 '12 at 14:20
2  
@gung thank you, you are right. I have edited that language out of my answer as I think it was inflammatory and unnecessary. –  Joshua Jul 18 '12 at 14:52
add comment

You shouldn't drop the intercept, regardless of whether you are likely or not to ever see all the explanatory variables having values of zero.

There's a good answer to a very similar question here.

If you remove the intercept then the other estimates all become biased. Even if the true value of the intercept is approximately zero (which is all you can conclude from your data), you are messing around with the slopes if you force it to be exactly zero.

UNLESS - you are measuring something with a very clear and obvious physical model that demands intercept be zero (eg you have height, width and length of a rectangular prism as explanatory variables and the response variable is volume with some measurement error). If your response variable is value of the house, you definitely need to leave the intercept in.

share|improve this answer
add comment

There are good answers here. Two small things:

  1. Regarding a higher $R^2$ when the intercept is dropped, you should read this excellent answer by @cardinal.
  2. Several people make the point that you should be certain the intercept must be 0 (for theoretical reasons) before dropping it, and not just that it isn't 'significant'. I think that's right, but it's not the whole story. You also need to know that the true data generating function is perfectly linear throughout the range of $X$ that you are working with and all the way down to 0. Remember that it is always possible that the function is approximately linear within your data, but actually slightly curving. It may be quite reasonable to treat the function as though it were linear within the range of your observations, even if it isn't perfectly so, but if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0.
share|improve this answer
    
@AdamO makes a similar point to #2 here: positive linear regression coefficient, but develops the idea much more fully. –  gung Mar 12 at 18:20
add comment

OK, so you've changed the question a LOT

You can leave out the intercept when you know it's 0. That's it. And no, you can't do it because it's not significantly different from 0, you have to know it's 0 or your residuals are biased. And, in that case it is 0 so it won't make any difference if you leave it out... therefore, never leave it out.

The finding you have with $R^2$ suggests the data are not linear. And, given that you had area as a predictor that particular one is probably definitely not linear. You could transform the predictor to fix that.

share|improve this answer
    
What about when we wish to test for cointegration using Engle/Granger 2-step? en.wikipedia.org/wiki/Cointegration –  Jase Dec 15 '12 at 11:42
add comment

Full revision of my thoughts. Indeed dropping the intercept will cause a bias problem.

Have you considered centering your data so an intercept would have some meaning and avoid explaining how some (unreasonable) values could give negative values? If you adjust all three explanatory variables by subtract the mean sqrft, mean lotsize and mean bath, then the intercept will now indicate the value (of a house?) with average sdrft, lotsize, and baths.

This centering will not change the relative relationship of the independent variables. So, fitting the model on the centered data will still find baths as insignificant. Refit the model without the bath included. You may still get a large p-value for the intercept, but it should be included and you will have a model of the form y=a+b(sqrft)+c(lotsize).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.