117
$\begingroup$

I am running linear regression models and wondering what the conditions are for removing the intercept term.

In comparing results from two different regressions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?

$\endgroup$
  • 1
    $\begingroup$ @chi thanks for editing my question. are there things that I should be clarifying or rewording in any future questions? $\endgroup$ – analyticsPierce Mar 7 '11 at 15:00
  • 3
    $\begingroup$ Your question is well stated. @chl kindly improved some formatting, that's all. It involved TeXifying the "R^2" (it was turned into $\$$R^2$\$$, which renders as $R^2$). $\endgroup$ – whuber Mar 7 '11 at 15:47
  • 1
    $\begingroup$ What would the intercept mean in your model? From the information in your question, it seems it would be the expected value of your response when sqft=0 and lotsize=0 and baths=0. Is that ever going to occur in reality? $\endgroup$ – timbp Feb 27 '12 at 2:59
  • 1
    $\begingroup$ Instead of the y= a + b1 x1 + b2 x2 + b3x3, can I omit a? $\endgroup$ – Travis Feb 27 '12 at 3:03
  • 3
    $\begingroup$ NB: Some of these comments and replies address essentially the same question (framed in the context of a housing price regression) which was merged with this one as a duplicate. $\endgroup$ – whuber Feb 27 '12 at 4:51
87
$\begingroup$

The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult Brooks Introductory Econometrics for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.

For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is "much" higher.

Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).

$\endgroup$
  • 3
    $\begingroup$ -1 for "never", see example 1 of Joshuas' answer $\endgroup$ – Curious Dec 2 '13 at 20:43
  • 4
    $\begingroup$ @Curious, "never" is written with "unless" examples below just show the exceptions when it is legal to remove intercept. When you don't know the data generating process or theory, or are not forced to go through the origin by standardization or any other special model, keep it. Keeping intercept is like using the trash bin to collect all the distortions caused by linear approximation and other simplifications. P.S. practically the response shows that you read just shortest :) Thanks a lot to Joshua (+1) for the extended examples. $\endgroup$ – Dmitrij Celov May 6 '14 at 9:56
  • 3
    $\begingroup$ You missed the point of Joshua Example 1 and seem to still ignore it completely. In models with categorical covariate the removal of the intercept results in the same model with just different parametrization. This is a legitimate case when intercept can be removed. $\endgroup$ – Curious May 26 '14 at 15:04
  • 2
    $\begingroup$ @Curious, in Joshua example 1, you need to add a new dummy variable for the level of the categorical variable you previously considered as baseline, and this new dummy variable will take the value of the intercept, so you are NOT removing the intercept, just renaming it and reparameterizing the rest of the parameters of the categorical covariate. Therefore the argument of Dmitrij holds. $\endgroup$ – Rufo Apr 15 at 15:17
59
$\begingroup$

Removing the intercept is a different model, but there are plenty of examples where it is legitimate. Answers so far have already discussed in detail the example where the true intercept is 0. I will focus on a few examples where we may be interested in an atypical model parametrization.

Example 1: The ANOVA-style Model. For categorical variables, we typically create binary vectors encoding group membership. The standard regression model is parametrized as intercept + k - 1 dummy vectors. The intercept codes the expected value for the "reference" group, or the omitted vector, and the remaining vectors test the difference between each group and the reference. But in some cases, it may be useful to have each groups' expected value.

dat <- mtcars
dat$vs <- factor(dat$vs)

## intercept model: vs coefficient becomes difference
lm(mpg ~ vs + hp, data = dat)

Coefficients:
(Intercept)          vs1           hp  
   26.96300      2.57622     -0.05453  

## no intercept: two vs coefficients, conditional expectations for both groups
lm(mpg ~ 0 + vs + hp, data = dat)

Coefficients:
     vs0       vs1        hp  
26.96300  29.53922  -0.05453  

Example 2: The case of standardized data. In some cases, one may be working with standardized data. In this case, the intercept is 0 by design. I think a classic example of this was old style structural equation models or factor, which operated just on the covariance matrices of data. In the case below, it is probably a good idea to estimate the intercept anyway, if only to drop the additional degree of freedom (which you really should have lost anyway because the mean was estimated), but there are a handful of situations where by construction, means may be 0 (e.g., certain experiments where participants assign ratings, but are constrained to give out equal positives and negatives).

dat <- as.data.frame(scale(mtcars))

## intercept is 0 by design
lm(mpg ~ hp + wt, data = dat)

Coefficients:
(Intercept)           hp           wt  
  3.813e-17   -3.615e-01   -6.296e-01  

## leaving the intercept out    
lm(mpg ~ 0 + hp + wt, data = dat)

Coefficients:
     hp       wt  
-0.3615  -0.6296  

Example 3: Multivariate Models and Hidden Intercepts. This example is similar to the first in many ways. In this case, the data has been stacked so that two different variables are now in one long vector. A second variable encodes information about whether the response vector, y, belongs to mpg or disp. In this case, to get the separate intercepts for each outcome, you suppress the overall intercept and include both dummy vectors for measure. This is a sort of multivariate analysis. It is not typically done using lm() because you have repeated measures and should probably allow for the nonindepence. However, there are some interesting cases where this is necessary. For example when trying to do a mediation analysis with random effects, to get the full variance covariance matrix, you need both models estimated simultaneously, which can be done by stacking the data and some clever use of dummy vectors.

## stack data for multivariate analysis
dat <- reshape(mtcars, varying = c(1, 3), v.names = "y",
  timevar = "measure", times = c("mpg", "disp"), direction = "long")
dat$measure <- factor(dat$measure)

## two regressions with intercepts only
lm(cbind(mpg, disp) ~ 1, data = mtcars)

Coefficients:
             mpg     disp  
(Intercept)   20.09  230.72

## using the stacked data, measure is difference between outcome means
lm(y ~ measure, data = dat)

Coefficients:
(Intercept)   measurempg  
      230.7       -210.6  

## separate 'intercept' for each outcome
lm(y ~ 0 + measure, data = dat)

Coefficients:
measuredisp   measurempg  
     230.72        20.09  

I am not arguing that intercepts should generally be removed, but it is good to be flexible.

$\endgroup$
  • 7
    $\begingroup$ +1. I didn't take people to be rigidly saying 'never', but it's always nice to have another perspective & this is a very clear & thoughtful response. Welcome to CV, it'll be great to have you as part of the community. $\endgroup$ – gung Jul 18 '12 at 14:20
  • 3
    $\begingroup$ @gung thank you, you are right. I have edited that language out of my answer as I think it was inflammatory and unnecessary. $\endgroup$ – Joshua Jul 18 '12 at 14:52
  • 1
    $\begingroup$ @Joshua: Sorry that I am asking a question on a almost 2 year old post, but are there any references on your first example? I am thinking of running a no-intercept model on my data where the predictor variable is categorical, and I am interested in knowing whether each level is significantly different from 0. Thanks! $\endgroup$ – Alex May 2 '14 at 19:40
  • $\begingroup$ @Alex Any good regression text ought to do (chapter 8 of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences 3rd ed. covers this some)--- you just need it to talk about contrasts and how to dummy code categorical variables. One way to think about it is that you are estimating separate intercepts for each group, rather than leaving the intercept out. $\endgroup$ – Joshua May 10 '14 at 22:39
  • $\begingroup$ @Joshua , your first example has caused some confusion elsewhere on this site. My understanding here is that you are suggesting a handy trick to show parameter estimates without needing to mess with adding the value for the intercept, and that you are not suggesting that the usual course is to use a model with the intercept removed to conduct anova. In R, in just about every case, one would use a model with an intercept to conduct a traditional anova. $\endgroup$ – Sal Mangiafico Oct 2 '18 at 19:04
29
$\begingroup$

There are good answers here. Two small things:

  1. Regarding a higher $R^2$ when the intercept is dropped, you should read this excellent answer by @cardinal. (In short, statistical software sometimes uses a different definition for $R^2$ when the intercept is forced to 0. So the reported $R^2$ for models with and without an intercept might simply not be comparable.)
  2. Several people make the point that you should be certain the intercept must be 0 (for theoretical reasons) before dropping it, and not just that it isn't 'significant'. I think that's right, but it's not the whole story. You also need to know that the true data generating function is perfectly linear throughout the range of $X$ that you are working with and all the way down to 0. Remember that it is always possible that the function is approximately linear within your data, but actually slightly curving. It may be quite reasonable to treat the function as though it were linear within the range of your observations, even if it isn't perfectly so, but if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0.
$\endgroup$
14
$\begingroup$

You shouldn't drop the intercept, regardless of whether you are likely or not to ever see all the explanatory variables having values of zero.

There's a good answer to a very similar question here.

If you remove the intercept then the other estimates all become biased. Even if the true value of the intercept is approximately zero (which is all you can conclude from your data), you are messing around with the slopes if you force it to be exactly zero.

UNLESS - you are measuring something with a very clear and obvious physical model that demands intercept be zero (eg you have height, width and length of a rectangular prism as explanatory variables and the response variable is volume with some measurement error). If your response variable is value of the house, you definitely need to leave the intercept in.

$\endgroup$
  • 1
    $\begingroup$ Can you please explain, why we need the intercept for house price prediction? why would all the X be zero for any house? $\endgroup$ – Elf Apr 2 at 6:27
10
$\begingroup$

OK, so you've changed the question a LOT

You can leave out the intercept when you know it's 0. That's it. And no, you can't do it because it's not significantly different from 0, you have to know it's 0 or your residuals are biased. And, in that case it is 0 so it won't make any difference if you leave it out... therefore, never leave it out.

The finding you have with $R^2$ suggests the data are not linear. And, given that you had area as a predictor that particular one is probably definitely not linear. You could transform the predictor to fix that.

$\endgroup$
5
$\begingroup$

Most multiple regression models include a constant term (i.e., the intercept), since this ensures that the model will be unbiased--i.e., the mean of the residuals will be exactly zero. (The coefficients in a regression model are estimated by least squares--i.e., minimizing the mean squared error. Now, the mean squared error is equal to the variance of the errors plus the square of their mean: this is a mathematical identity. Changing the value of the constant in the model changes the mean of the errors but doesn't affect the variance. Hence, if the sum of squared errors is to be minimized, the constant must be chosen such that the mean of the errors is zero.)

In a simple regression model, the constant represents the Y-intercept of the regression line, in unstandardized form. In a multiple regression model, the constant represents the value that would be predicted for the dependent variable if all the independent variables were simultaneously equal to zero--a situation which may not physically or economically meaningful. If you are not particularly interested in what would happen if all the independent variables were simultaneously zero, then you normally leave the constant in the model regardless of its statistical significance. In addition to ensuring that the in-sample errors are unbiased, the presence of the constant allows the regression line to "seek its own level" and provide the best fit to data which may only be locally linear.

However, in rare cases you may wish to exclude the constant from the model. This is a model-fitting option in the regression procedure in any software package, and it is sometimes referred to as regression through the origin, or RTO for short. Usually, this will be done only if:

  1. it is possible to imagine the independent variables all assuming the value zero simultaneously, and you feel that in this case it should logically follow that the dependent variable will also be equal to zero; or else
  2. the constant is redundant with the set of independent variables you wish to use.

An example of case (1) would be a model in which all variables--dependent and independent--represented first differences of other time series. If you are regressing the first difference of Y on the first difference of X, you are directly predicting changes in Y as a linear function of changes in X, without reference to the current levels of the variables. In this case it might be reasonable (although not required) to assume that Y should be unchanged, on the average, whenever X is unchanged--i.e., that Y should not have an upward or downward trend in the absence of any change in the level of X.

An example of case (2) would be a situation in which you wish to use a full set of seasonal indicator variables--e.g., you are using quarterly data, and you wish to include variables Q1, Q2, Q3, and Q4 representing additive seasonal effects. Thus, Q1 might look like 1 0 0 0 1 0 0 0 ..., Q2 would look like 0 1 0 0 0 1 0 0 ..., and so on. You could not use all four of these and a constant in the same model, since Q1+Q2+Q3+Q4 = 1 1 1 1 1 1 1 1 . . . . , which is the same as a constant term. I.e., the five variables Q1, Q2, Q3, Q4, and CONSTANT are not linearly independent: any one of them can be expressed as a linear combination of the other four. A technical prerequisite for fitting a linear regression model is that the independent variables must be linearly independent; otherwise the least-squares coefficients cannot be determined uniquely, and we say the regression "fails."

A word of warning: R-squared and the F statistic do not have the same meaning in an RTO model as they do in an ordinary regression model, and they are not calculated in the same way by all software. See this article for some caveats. You should not try to compare R-squared between models that do and do not include a constant term, although it is OK to compare the standard error of the regression.

Note that the term "independent" is used in (at least) three different ways in regression jargon: any single variable may be called an independent variable if it is being used as a predictor, rather than as the predictee. A group of variables is linearly independent if no one of them can be expressed exactly as a linear combination of the others. A pair of variables is said to be statistically independent if they are not only linearly independent but also utterly uninformative with respect to each other. In a regression model, you want your dependent variable to be statistically dependent on the independent variables, which must be linearly (but not necessarily statistically) independent among themselves.

$\endgroup$
  • 2
    $\begingroup$ What article are you referring to? $\endgroup$ – gung Dec 10 '14 at 16:31
2
$\begingroup$

Full revision of my thoughts. Indeed dropping the intercept will cause a bias problem.

Have you considered centering your data so an intercept would have some meaning and avoid explaining how some (unreasonable) values could give negative values? If you adjust all three explanatory variables by subtract the mean sqrft, mean lotsize and mean bath, then the intercept will now indicate the value (of a house?) with average sdrft, lotsize, and baths.

This centering will not change the relative relationship of the independent variables. So, fitting the model on the centered data will still find baths as insignificant. Refit the model without the bath included. You may still get a large p-value for the intercept, but it should be included and you will have a model of the form y=a+b(sqrft)+c(lotsize).

$\endgroup$
1
$\begingroup$

I just spent some time answering a similar question posted by someone else, but it was closed. There are some great answers here, but the answer I provide is a bit simpler. It might be more suited to people who have a weak understanding of regression.

Q1: How do I interpret the intercept in my model?

In regression models, the goal is to minimise the amount of unexplained variance in an outcome variable:

y = b0 + b1⋅x + ϵ

where y is the predicted value of your outcome measure (e.g., log_blood_hg), b0 is the intercept, b1 is the slope, x is a predictor variable, and ϵ is residual error.

The intercept (b0) is the predicted mean value of y when all x = 0. In other words, it's the baseline value of y, before you've used any variables (e.g., species) to further minimise or explain the variance in log_blood_hg.

By adding a slope (which estimates how a one-unit increase/decrease in log_blood_hg changes with a one unit increase in x, e.g., species), we add to what we already know about the outcome variable, which is its baseline value (i.e. intercept), based on change in another variable.

Q2: When is it appropriate to include or not include the intercept, especially in regards to the fact that the models give very different results?

For simple models like this, it's never really appropriate to drop the intercept.

The models give different results when you drop the intercept because rather than grounding the slope in the baseline value of Y, it is forced to go through the origin of y, which is 0. Therefore, the slope gets steeper (i.e. more powerful and significant) because you've forced the line through the origin, not because it does a better job of minimizing the variance in y. In other words, you've artificially created a model which minimizes the variance in y by removing the intercept, or the initial grounding point for your model.

There are cases where removing the intercept is appropriate - such as when describing a phenomenon with a 0-intercept. You can read about that here, as well as more reasons why removing an intercept isn't a good idea.

$\endgroup$
0
$\begingroup$

Short answer: (almost) NEVER. In the linear regression model $$ y = \alpha + \beta x + \epsilon $$, if you set $\alpha=0$, then you say that you KNOW that the expected value of $y$ given $x=0$ is zero. You almost never know that.

$R^2$ becomes higher without intercept, not because the model is better, but because the definition of $R^2$ used is another one! $R^2$ is an expression of a comparison of the estimated model with some standard model, expressed as reduction in sum of squares compared to sum of squares with the standard model. In the model with intercept, the comparison sum of squares is around the mean. Without intercept, it is around zero! The last one is usually much higher, so it easier to get a large reduction in sum of squares.

Conclusion: DO NOT LEAVE THE INTERCEPT OUT OF THE MODEL (unless you really, really know what you are doing).

Some exceptions: One exception is a regression representing a one-way ANOVA with dummies for ALL the factor levels (usually one is left out) (but that is only seemingly an exception, the constant vector 1 is in the column space of the model matrix $X$.) Otherwise, such as physical relationships $s=v t$ where there are no constant. But even then, if the model is only approximate (speed is not really constant), it might be better to leave in a constant even if it cannot be interpreted.

There are also special models which leave out the intercept. One example is paired data, twin studies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.