Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am running linear regression models and wondering what the conditions are for removing the intercept term of lm()?

In comparing results from two different lm functions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?

Any advice is appreciated.

share|improve this question
    
@chi thanks for editing my question. are there things that I should be clarifying or rewording in any future questions? –  analyticsPierce Mar 7 '11 at 15:00
    
Your question is well stated. @chl kindly improved some formatting, that's all. It involved TeXifying the "R^2" (it was turned into $\$$R^2$\$$, which renders as $R^2$). –  whuber Mar 7 '11 at 15:47
    
What would the intercept mean in your model? From the information in your question, it seems it would be the expected value of your response when sqft=0 and lotsize=0 and baths=0. Is that ever going to occur in reality? –  timbp Feb 27 '12 at 2:59
    
the intercept has no meaning. so when writing up a formula for the expected value of a house, can I leave this out? –  Travis Feb 27 '12 at 3:02
1  
NB: Some of these comments and replies address essentially the same question (framed in the context of a housing price regression) which was merged with this one as a duplicate. –  whuber Feb 27 '12 at 4:51

7 Answers 7

up vote 31 down vote accepted

The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult Brooks Introductory Econometrics for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.

For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is "much" higher.

Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).

share|improve this answer
    
Great summary, thank you for the insight. –  analyticsPierce Mar 7 '11 at 15:06
    
-1 for "never", see example 1 of Joshuas' answer –  Curious Dec 2 '13 at 20:43
    
@Curious, "never" is written with "unless" examples below just show the exceptions when it is legal to remove intercept. When you don't know the data generating process or theory, or are not forced to go through the origin by standardization or any other special model, keep it. Keeping intercept is like using the trash bin to collect all the distortions caused by linear approximation and other simplifications. P.S. practically the response shows that you read just shortest :) Thanks a lot to Joshua (+1) for the extended examples. –  Dmitrij Celov May 6 at 9:56
    
You missed the point of Joshua Example 1 and seem to still ignore it completely. In models with categorical covariate the removal of the intercept results in the same model with just different parametrization. This is a legitimate case when intercept can be removed. –  Curious May 26 at 15:04

Removing the intercept is a different model, but there are plenty of examples where it is legitimate. Answers so far have already discussed in detail the example where the true intercept is 0. I will focus on a few examples where we may be interested in an atypical model parametrization.

Example 1: The ANOVA-style Model. For categorical variables, we typically create binary vectors encoding group membership. The standard regression model is parametrized as intercept + k - 1 dummy vectors. The intercept codes the expected value for the "reference" group, or the omitted vector, and the remaining vectors test the difference between each group and the reference. But in some cases, it may be useful to have each groups' expected value.

dat <- mtcars
dat$vs <- factor(dat$vs)

## intercept model: vs coefficient becomes difference
lm(mpg ~ vs + hp, data = dat)

Coefficients:
(Intercept)          vs1           hp  
   26.96300      2.57622     -0.05453  

## no intercept: two vs coefficients, conditional expectations for both groups
lm(mpg ~ 0 + vs + hp, data = dat)

Coefficients:
     vs0       vs1        hp  
26.96300  29.53922  -0.05453  

Example 2: The case of standardized data. In some cases, one may be working with standardized data. In this case, the intercept is 0 by design. I think a classic example of this was old style structural equation models or factor, which operated just on the covariance matrices of data. In the case below, it is probably a good idea to estimate the intercept anyway, if only to drop the additional degree of freedom (which you really should have lost anyway because the mean was estimated), but there are a handful of situations where by construction, means may be 0 (e.g., certain experiments where participants assign ratings, but are constrained to give out equal positives and negatives).

dat <- as.data.frame(scale(mtcars))

## intercept is 0 by design
lm(mpg ~ hp + wt, data = dat)

Coefficients:
(Intercept)           hp           wt  
  3.813e-17   -3.615e-01   -6.296e-01  

## leaving the intercept out    
lm(mpg ~ 0 + hp + wt, data = dat)

Coefficients:
     hp       wt  
-0.3615  -0.6296  

Example 3: Multivariate Models and Hidden Intercepts. This example is similar to the first in many ways. In this case, the data has been stacked so that two different variables are now in one long vector. A second variable encodes information about whether the response vector, y, belongs to mpg or disp. In this case, to get the separate intercepts for each outcome, you suppress the overall intercept and include both dummy vectors for measure. This is a sort of multivariate analysis. It is not typically done using lm() because you have repeated measures and should probably allow for the nonindepence. However, there are some interesting cases where this is necessary. For example when trying to do a mediation analysis with random effects, to get the full variance covariance matrix, you need both models estimated simultaneously, which can be done by stacking the data and some clever use of dummy vectors.

## stack data for multivariate analysis
dat <- reshape(mtcars, varying = c(1, 3), v.names = "y",
  timevar = "measure", times = c("mpg", "disp"), direction = "long")
dat$measure <- factor(dat$measure)

## two regressions with intercepts only
lm(cbind(mpg, disp) ~ 1, data = mtcars)

Coefficients:
             mpg     disp  
(Intercept)   20.09  230.72

## using the stacked data, measure is difference between outcome means
lm(y ~ measure, data = dat)

Coefficients:
(Intercept)   measurempg  
      230.7       -210.6  

## separate 'intercept' for each outcome
lm(y ~ 0 + measure, data = dat)

Coefficients:
measuredisp   measurempg  
     230.72        20.09  

I am not arguing that intercepts should generally be removed, but it is good to be flexible.

share|improve this answer
3  
+1. I didn't take people to be rigidly saying 'never', but it's always nice to have another perspective & this is a very clear & thoughtful response. Welcome to CV, it'll be great to have you as part of the community. –  gung Jul 18 '12 at 14:20
2  
@gung thank you, you are right. I have edited that language out of my answer as I think it was inflammatory and unnecessary. –  Joshua Jul 18 '12 at 14:52
    
@Joshua: Sorry that I am asking a question on a almost 2 year old post, but are there any references on your first example? I am thinking of running a no-intercept model on my data where the predictor variable is categorical, and I am interested in knowing whether each level is significantly different from 0. Thanks! –  Alex May 2 at 19:40
    
@Alex Any good regression text ought to do (chapter 8 of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences 3rd ed. covers this some)--- you just need it to talk about contrasts and how to dummy code categorical variables. One way to think about it is that you are estimating separate intercepts for each group, rather than leaving the intercept out. –  Joshua May 10 at 22:39

You shouldn't drop the intercept, regardless of whether you are likely or not to ever see all the explanatory variables having values of zero.

There's a good answer to a very similar question here.

If you remove the intercept then the other estimates all become biased. Even if the true value of the intercept is approximately zero (which is all you can conclude from your data), you are messing around with the slopes if you force it to be exactly zero.

UNLESS - you are measuring something with a very clear and obvious physical model that demands intercept be zero (eg you have height, width and length of a rectangular prism as explanatory variables and the response variable is volume with some measurement error). If your response variable is value of the house, you definitely need to leave the intercept in.

share|improve this answer

There are good answers here. Two small things:

  1. Regarding a higher $R^2$ when the intercept is dropped, you should read this excellent answer by @cardinal.
  2. Several people make the point that you should be certain the intercept must be 0 (for theoretical reasons) before dropping it, and not just that it isn't 'significant'. I think that's right, but it's not the whole story. You also need to know that the true data generating function is perfectly linear throughout the range of $X$ that you are working with and all the way down to 0. Remember that it is always possible that the function is approximately linear within your data, but actually slightly curving. It may be quite reasonable to treat the function as though it were linear within the range of your observations, even if it isn't perfectly so, but if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0.
share|improve this answer
    
@AdamO makes a similar point to #2 here: positive linear regression coefficient, but develops the idea much more fully. –  gung Mar 12 at 18:20

OK, so you've changed the question a LOT

You can leave out the intercept when you know it's 0. That's it. And no, you can't do it because it's not significantly different from 0, you have to know it's 0 or your residuals are biased. And, in that case it is 0 so it won't make any difference if you leave it out... therefore, never leave it out.

The finding you have with $R^2$ suggests the data are not linear. And, given that you had area as a predictor that particular one is probably definitely not linear. You could transform the predictor to fix that.

share|improve this answer
    
What about when we wish to test for cointegration using Engle/Granger 2-step? en.wikipedia.org/wiki/Cointegration –  Jase Dec 15 '12 at 11:42

Full revision of my thoughts. Indeed dropping the intercept will cause a bias problem.

Have you considered centering your data so an intercept would have some meaning and avoid explaining how some (unreasonable) values could give negative values? If you adjust all three explanatory variables by subtract the mean sqrft, mean lotsize and mean bath, then the intercept will now indicate the value (of a house?) with average sdrft, lotsize, and baths.

This centering will not change the relative relationship of the independent variables. So, fitting the model on the centered data will still find baths as insignificant. Refit the model without the bath included. You may still get a large p-value for the intercept, but it should be included and you will have a model of the form y=a+b(sqrft)+c(lotsize).

share|improve this answer

Most multiple regression models include a constant term (i.e., the intercept), since this ensures that the model will be unbiased--i.e., the mean of the residuals will be exactly zero. (The coefficients in a regression model are estimated by least squares--i.e., minimizing the mean squared error. Now, the mean squared error is equal to the variance of the errors plus the square of their mean: this is a mathematical identity. Changing the value of the constant in the model changes the mean of the errors but doesn't affect the variance. Hence, if the sum of squared errors is to be minimized, the constant must be chosen such that the mean of the errors is zero.)

In a simple regression model, the constant represents the Y-intercept of the regression line, in unstandardized form. In a multiple regression model, the constant represents the value that would be predicted for the dependent variable if all the independent variables were simultaneously equal to zero--a situation which may not physically or economically meaningful. If you are not particularly interested in what would happen if all the independent variables were simultaneously zero, then you normally leave the constant in the model regardless of its statistical significance. In addition to ensuring that the in-sample errors are unbiased, the presence of the constant allows the regression line to "seek its own level" and provide the best fit to data which may only be locally linear.

However, in rare cases you may wish to exclude the constant from the model. This is a model-fitting option in the regression procedure in any software package, and it is sometimes referred to as regression through the origin, or RTO for short. Usually, this will be done only if:

  1. it is possible to imagine the independent variables all assuming the value zero simultaneously, and you feel that in this case it should logically follow that the dependent variable will also be equal to zero; or else
  2. the constant is redundant with the set of independent variables you wish to use.

An example of case (1) would be a model in which all variables--dependent and independent--represented first differences of other time series. If you are regressing the first difference of Y on the first difference of X, you are directly predicting changes in Y as a linear function of changes in X, without reference to the current levels of the variables. In this case it might be reasonable (although not required) to assume that Y should be unchanged, on the average, whenever X is unchanged--i.e., that Y should not have an upward or downward trend in the absence of any change in the level of X.

An example of case (2) would be a situation in which you wish to use a full set of seasonal indicator variables--e.g., you are using quarterly data, and you wish to include variables Q1, Q2, Q3, and Q4 representing additive seasonal effects. Thus, Q1 might look like 1 0 0 0 1 0 0 0 ..., Q2 would look like 0 1 0 0 0 1 0 0 ..., and so on. You could not use all four of these and a constant in the same model, since Q1+Q2+Q3+Q4 = 1 1 1 1 1 1 1 1 . . . . , which is the same as a constant term. I.e., the five variables Q1, Q2, Q3, Q4, and CONSTANT are not linearly independent: any one of them can be expressed as a linear combination of the other four. A technical prerequisite for fitting a linear regression model is that the independent variables must be linearly independent; otherwise the least-squares coefficients cannot be determined uniquely, and we say the regression "fails."

A word of warning: R-squared and the F statistic do not have the same meaning in an RTO model as they do in an ordinary regression model, and they are not calculated in the same way by all software. See this article for some caveats. You should not try to compare R-squared between models that do and do not include a constant term, although it is OK to compare the standard error of the regression.

Note that the term "independent" is used in (at least) three different ways in regression jargon: any single variable may be called an independent variable if it is being used as a predictor, rather than as the predictee. A group of variables is linearly independent if no one of them can be expressed exactly as a linear combination of the others. A pair of variables is said to be statistically independent if they are not only linearly independent but also utterly uninformative with respect to each other. In a regression model, you want your dependent variable to be statistically dependent on the independent variables, which must be linearly (but not necessarily statistically) independent among themselves.

share|improve this answer
1  
What article are you referring to? –  gung Dec 10 at 16:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.