181
$\begingroup$

I am running linear regression models and wondering what the conditions are for removing the intercept term.

In comparing results from two different regressions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?

$\endgroup$
5
  • 1
    $\begingroup$ What would the intercept mean in your model? From the information in your question, it seems it would be the expected value of your response when sqft=0 and lotsize=0 and baths=0. Is that ever going to occur in reality? $\endgroup$
    – timbp
    Feb 27, 2012 at 2:59
  • $\begingroup$ the intercept has no meaning. so when writing up a formula for the expected value of a house, can I leave this out? $\endgroup$
    – Travis
    Feb 27, 2012 at 3:02
  • 1
    $\begingroup$ Instead of the y= a + b1 x1 + b2 x2 + b3x3, can I omit a? $\endgroup$
    – Travis
    Feb 27, 2012 at 3:03
  • 3
    $\begingroup$ NB: Some of these comments and replies address essentially the same question (framed in the context of a housing price regression) which was merged with this one as a duplicate. $\endgroup$
    – whuber
    Feb 27, 2012 at 4:51
  • 1
    $\begingroup$ A side point: $R^2$ does not have such good properties when you don't include an intercept in your model, so it's not the best statistic to use for your comparison. $\endgroup$
    – Charlie
    Feb 27, 2012 at 7:43

10 Answers 10

124
$\begingroup$

The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult Brooks Introductory Econometrics for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.

For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is "much" higher.

Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).

$\endgroup$
4
  • 11
    $\begingroup$ -1 for "never", see example 1 of Joshuas' answer $\endgroup$
    – Tomas
    Dec 2, 2013 at 20:43
  • 7
    $\begingroup$ @Curious, "never" is written with "unless" examples below just show the exceptions when it is legal to remove intercept. When you don't know the data generating process or theory, or are not forced to go through the origin by standardization or any other special model, keep it. Keeping intercept is like using the trash bin to collect all the distortions caused by linear approximation and other simplifications. P.S. practically the response shows that you read just shortest :) Thanks a lot to Joshua (+1) for the extended examples. $\endgroup$ May 6, 2014 at 9:56
  • 5
    $\begingroup$ You missed the point of Joshua Example 1 and seem to still ignore it completely. In models with categorical covariate the removal of the intercept results in the same model with just different parametrization. This is a legitimate case when intercept can be removed. $\endgroup$
    – Tomas
    May 26, 2014 at 15:04
  • 3
    $\begingroup$ @Curious, in Joshua example 1, you need to add a new dummy variable for the level of the categorical variable you previously considered as baseline, and this new dummy variable will take the value of the intercept, so you are NOT removing the intercept, just renaming it and reparameterizing the rest of the parameters of the categorical covariate. Therefore the argument of Dmitrij holds. $\endgroup$
    – Rufo
    Apr 15, 2019 at 15:17
90
$\begingroup$

Removing the intercept is a different model, but there are plenty of examples where it is legitimate. Answers so far have already discussed in detail the example where the true intercept is 0. I will focus on a few examples where we may be interested in an atypical model parametrization.

Example 1: The ANOVA-style Model. For categorical variables, we typically create binary vectors encoding group membership. The standard regression model is parametrized as intercept + k - 1 dummy vectors. The intercept codes the expected value for the "reference" group, or the omitted vector, and the remaining vectors test the difference between each group and the reference. But in some cases, it may be useful to have each groups' expected value.

dat <- mtcars
dat$vs <- factor(dat$vs)

## intercept model: vs coefficient becomes difference
lm(mpg ~ vs + hp, data = dat)

Coefficients:
(Intercept)          vs1           hp  
   26.96300      2.57622     -0.05453  

## no intercept: two vs coefficients, 
## conditional expectations for both groups
lm(mpg ~ 0 + vs + hp, data = dat)

Coefficients:
     vs0       vs1        hp  
26.96300  29.53922  -0.05453  

Example 2: The case of standardized data. In some cases, one may be working with standardized data. In this case, the intercept is 0 by design. I think a classic example of this was old style structural equation models or factor, which operated just on the covariance matrices of data. In the case below, it is probably a good idea to estimate the intercept anyway, if only to drop the additional degree of freedom (which you really should have lost anyway because the mean was estimated), but there are a handful of situations where by construction, means may be 0 (e.g., certain experiments where participants assign ratings, but are constrained to give out equal positives and negatives).

dat <- as.data.frame(scale(mtcars))

## intercept is 0 by design
lm(mpg ~ hp + wt, data = dat)

Coefficients:
(Intercept)           hp           wt  
  3.813e-17   -3.615e-01   -6.296e-01  

## leaving the intercept out    
lm(mpg ~ 0 + hp + wt, data = dat)

Coefficients:
     hp       wt  
-0.3615  -0.6296  

Example 3: Multivariate Models and Hidden Intercepts. This example is similar to the first in many ways. In this case, the data has been stacked so that two different variables are now in one long vector. A second variable encodes information about whether the response vector, y, belongs to mpg or disp. In this case, to get the separate intercepts for each outcome, you suppress the overall intercept and include both dummy vectors for measure. This is a sort of multivariate analysis. It is not typically done using lm() because you have repeated measures and should probably allow for the nonindepence. However, there are some interesting cases where this is necessary. For example when trying to do a mediation analysis with random effects, to get the full variance covariance matrix, you need both models estimated simultaneously, which can be done by stacking the data and some clever use of dummy vectors.

## stack data for multivariate analysis
dat <- reshape(mtcars, varying = c(1, 3), v.names = "y",
  timevar = "measure", times = c("mpg", "disp"), 
             direction = "long")
dat$measure <- factor(dat$measure)

## two regressions with intercepts only
lm(cbind(mpg, disp) ~ 1, data = mtcars)

Coefficients:
             mpg     disp  
(Intercept)   20.09  230.72

## using the stacked data, measure is difference 
## between outcome means
lm(y ~ measure, data = dat)

Coefficients:
(Intercept)   measurempg  
      230.7       -210.6  

## separate 'intercept' for each outcome
lm(y ~ 0 + measure, data = dat)

Coefficients:
measuredisp   measurempg  
     230.72        20.09  

I am not arguing that intercepts should generally be removed, but it is good to be flexible.

$\endgroup$
5
  • 9
    $\begingroup$ +1. I didn't take people to be rigidly saying 'never', but it's always nice to have another perspective & this is a very clear & thoughtful response. Welcome to CV, it'll be great to have you as part of the community. $\endgroup$ Jul 18, 2012 at 14:20
  • 2
    $\begingroup$ @Joshua: Sorry that I am asking a question on a almost 2 year old post, but are there any references on your first example? I am thinking of running a no-intercept model on my data where the predictor variable is categorical, and I am interested in knowing whether each level is significantly different from 0. Thanks! $\endgroup$
    – Alex
    May 2, 2014 at 19:40
  • 1
    $\begingroup$ @Alex Any good regression text ought to do (chapter 8 of Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences 3rd ed. covers this some)--- you just need it to talk about contrasts and how to dummy code categorical variables. One way to think about it is that you are estimating separate intercepts for each group, rather than leaving the intercept out. $\endgroup$
    – Joshua
    May 10, 2014 at 22:39
  • 1
    $\begingroup$ @Joshua , your first example has caused some confusion elsewhere on this site. My understanding here is that you are suggesting a handy trick to show parameter estimates without needing to mess with adding the value for the intercept, and that you are not suggesting that the usual course is to use a model with the intercept removed to conduct anova. In R, in just about every case, one would use a model with an intercept to conduct a traditional anova. $\endgroup$ Oct 2, 2018 at 19:04
  • $\begingroup$ @SalMangiafico Yes, that's correct. I'm not advocating removing the intercept as a standard model approach. $\endgroup$
    – Joshua
    Aug 19, 2020 at 7:35
41
$\begingroup$

There are good answers here. Two small things:

  1. Regarding a higher $R^2$ when the intercept is dropped, you should read this excellent answer by @cardinal. (In short, statistical software sometimes uses a different definition for $R^2$ when the intercept is forced to 0. So the reported $R^2$ for models with and without an intercept might simply not be comparable.)
  2. Several people make the point that you should be certain the intercept must be 0 (for theoretical reasons) before dropping it, and not just that it isn't 'significant'. I think that's right, but it's not the whole story. You also need to know that the true data generating function is perfectly linear throughout the range of $X$ that you are working with and all the way down to 0. Remember that it is always possible that the function is approximately linear within your data, but actually slightly curving. It may be quite reasonable to treat the function as though it were linear within the range of your observations, even if it isn't perfectly so, but if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0.
$\endgroup$
7
  • 2
    $\begingroup$ @AdamO makes a similar point to #2 here: positive linear regression coefficient, but develops the idea much more fully. $\endgroup$ Mar 12, 2014 at 18:20
  • 1
    $\begingroup$ "if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0" Wouldn't this be the case for many other terms as well? For example, the function $y=x^3+x+1$ has no quadratic term, but the model family $y=mx+b$ will fit worse than one without a quadratic term. This just seems to be a symptom of underfitting generally, and not a unique property of the intercept term. $\endgroup$
    – Him
    Apr 27, 2020 at 17:27
  • $\begingroup$ @Scott, when you have a cubic term, the $x^2$ is a kind of intercept (see here: Does it make sense to add a quadratic term but not the linear term to a model?). $\endgroup$ Apr 28, 2020 at 4:29
  • $\begingroup$ "even if it isn't perfectly so, but if it isn't and you drop the intercept you will end up with a worse approximation to the underlying function even if the true intercept is 0." If dropping the intercept would result in a lot worse estimate/prediction then the intercept term will often not be insignificant. When the intercept term is insignificant then this means that the noise is probably larger than the value of the intercept term. The estimate of the intercept might fit the noise rather than the true underlying intercept and this can increase the expected squared error. $\endgroup$ Sep 14, 2022 at 8:45
  • $\begingroup$ @SextusEmpiricus statistical significance isn't the same as predictive accuracy. It's a fallacy to assume that whatever is significant will enhance your out of sample predictive accuracy & that whatever is nonsignificant will lead to overfitting & detract from predictive accuracy. $\endgroup$ Sep 14, 2022 at 11:26
19
$\begingroup$

You shouldn't drop the intercept, regardless of whether you are likely or not to ever see all the explanatory variables having values of zero.

If you remove the intercept then the other estimates all become biased. Even if the true value of the intercept is approximately zero (which is all you can conclude from your data), you are messing around with the slopes if you force it to be exactly zero.

UNLESS - you are measuring something with a very clear and obvious physical model that demands intercept be zero (eg you have height, width and length of a rectangular prism as explanatory variables and the response variable is volume with some measurement error). If your response variable is value of the house, you definitely need to leave the intercept in.

$\endgroup$
4
  • 2
    $\begingroup$ Can you please explain, why we need the intercept for house price prediction? why would all the X be zero for any house? $\endgroup$
    – Elf
    Apr 2, 2019 at 6:27
  • 1
    $\begingroup$ A bit late here, but I'm guessing because it's unlikely the value of a house would ever be 0, regardless of it's characteristics. $\endgroup$
    – Deruijter
    Feb 4, 2022 at 13:40
  • $\begingroup$ Land alone is worth something, so even a dimensionless house is valuable. $\endgroup$ Aug 31, 2022 at 18:55
  • 1
    $\begingroup$ "If you remove the intercept then the other estimates all become biased" This is not specific for the intercept. The same is true for other regressor terms. If we leave out other regressor terms that are approximately zero then we also introduce bias. It also doesn't matter so much because bias is not the main parameter that we care about. Often we find it more important to reduce the expectation of the square of the error or the expectation of some other non-linear cost function. $\endgroup$ Sep 14, 2022 at 8:39
16
$\begingroup$

OK, so you've changed the question a LOT

You can leave out the intercept when you know it's 0. That's it. And no, you can't do it because it's not significantly different from 0, you have to know it's 0 or your residuals are biased. And, in that case it is 0 so it won't make any difference if you leave it out... therefore, never leave it out.

The finding you have with $R^2$ suggests the data are not linear. And, given that you had area as a predictor that particular one is probably definitely not linear. You could transform the predictor to fix that.

$\endgroup$
4
  • 2
    $\begingroup$ What about when we wish to test for cointegration using Engle/Granger 2-step? en.wikipedia.org/wiki/Cointegration $\endgroup$
    – Jase
    Dec 15, 2012 at 11:42
  • 1
    $\begingroup$ "you have to know it's 0 or your residuals are biased" many people miss this crucial point and say that the beta will be biased, but that's wrong because the betas are not in any way dependent on the intercept. $\endgroup$
    – baibo
    Mar 3, 2022 at 7:34
  • $\begingroup$ @baibo the betas/coefficients might become biased when the regressors correlate with the (removed) intercept term. When you fit a model $y = bx$ while the true model is $y=a+bx$ then the estimate of $b$ will become biased. An obvious example is in the image here: stats.stackexchange.com/a/325395/164061 $\endgroup$ Sep 14, 2022 at 8:31
  • $\begingroup$ "you have to know it's 0 or your residuals are biased" What is the problem with this? Do we care about this bias when the goal is to make a best prediction, for instance based on reducing the expectation of the squared error? $\endgroup$ Sep 14, 2022 at 8:34
11
$\begingroup$

Most multiple regression models include a constant term (i.e., the intercept), since this ensures that the model will be unbiased--i.e., the mean of the residuals will be exactly zero. (The coefficients in a regression model are estimated by least squares--i.e., minimizing the mean squared error. Now, the mean squared error is equal to the variance of the errors plus the square of their mean: this is a mathematical identity. Changing the value of the constant in the model changes the mean of the errors but doesn't affect the variance. Hence, if the sum of squared errors is to be minimized, the constant must be chosen such that the mean of the errors is zero.)

In a simple regression model, the constant represents the Y-intercept of the regression line, in unstandardized form. In a multiple regression model, the constant represents the value that would be predicted for the dependent variable if all the independent variables were simultaneously equal to zero--a situation which may not physically or economically meaningful. If you are not particularly interested in what would happen if all the independent variables were simultaneously zero, then you normally leave the constant in the model regardless of its statistical significance. In addition to ensuring that the in-sample errors are unbiased, the presence of the constant allows the regression line to "seek its own level" and provide the best fit to data which may only be locally linear.

However, in rare cases you may wish to exclude the constant from the model. This is a model-fitting option in the regression procedure in any software package, and it is sometimes referred to as regression through the origin, or RTO for short. Usually, this will be done only if:

  1. it is possible to imagine the independent variables all assuming the value zero simultaneously, and you feel that in this case it should logically follow that the dependent variable will also be equal to zero; or else
  2. the constant is redundant with the set of independent variables you wish to use.

An example of case (1) would be a model in which all variables--dependent and independent--represented first differences of other time series. If you are regressing the first difference of Y on the first difference of X, you are directly predicting changes in Y as a linear function of changes in X, without reference to the current levels of the variables. In this case it might be reasonable (although not required) to assume that Y should be unchanged, on the average, whenever X is unchanged--i.e., that Y should not have an upward or downward trend in the absence of any change in the level of X.

An example of case (2) would be a situation in which you wish to use a full set of seasonal indicator variables--e.g., you are using quarterly data, and you wish to include variables Q1, Q2, Q3, and Q4 representing additive seasonal effects. Thus, Q1 might look like 1 0 0 0 1 0 0 0 ..., Q2 would look like 0 1 0 0 0 1 0 0 ..., and so on. You could not use all four of these and a constant in the same model, since Q1+Q2+Q3+Q4 = 1 1 1 1 1 1 1 1 . . . . , which is the same as a constant term. I.e., the five variables Q1, Q2, Q3, Q4, and CONSTANT are not linearly independent: any one of them can be expressed as a linear combination of the other four. A technical prerequisite for fitting a linear regression model is that the independent variables must be linearly independent; otherwise the least-squares coefficients cannot be determined uniquely, and we say the regression "fails."

A word of warning: R-squared and the F statistic do not have the same meaning in an RTO model as they do in an ordinary regression model, and they are not calculated in the same way by all software. See this article for some caveats. You should not try to compare R-squared between models that do and do not include a constant term, although it is OK to compare the standard error of the regression.

Note that the term "independent" is used in (at least) three different ways in regression jargon: any single variable may be called an independent variable if it is being used as a predictor, rather than as the predictee. A group of variables is linearly independent if no one of them can be expressed exactly as a linear combination of the others. A pair of variables is said to be statistically independent if they are not only linearly independent but also utterly uninformative with respect to each other. In a regression model, you want your dependent variable to be statistically dependent on the independent variables, which must be linearly (but not necessarily statistically) independent among themselves.

$\endgroup$
1
  • 2
    $\begingroup$ What article are you referring to? $\endgroup$ Dec 10, 2014 at 16:31
5
$\begingroup$

Short answer: (almost) NEVER. In the linear regression model $$ y = \alpha + \beta x + \epsilon $$, if you set $\alpha=0$, then you say that you KNOW that the expected value of $y$ given $x=0$ is zero. You almost never know that.

$R^2$ becomes higher without intercept, not because the model is better, but because the definition of $R^2$ used is another one! $R^2$ is an expression of a comparison of the estimated model with some standard model, expressed as reduction in sum of squares compared to sum of squares with the standard model. In the model with intercept, the comparison sum of squares is around the mean. Without intercept, it is around zero! The last one is usually much higher, so it easier to get a large reduction in sum of squares.

Conclusion: DO NOT LEAVE THE INTERCEPT OUT OF THE MODEL (unless you really, really know what you are doing).

Some exceptions: One exception is a regression representing a one-way ANOVA with dummies for ALL the factor levels (usually one is left out) (but that is only seemingly an exception, the constant vector 1 is in the column space of the model matrix $X$.)

Compositional data as predictors (but also only seemingly an exception) as an example see How to measure ML model performance across race via area level demographic info

Otherwise, such as physical relationships $s=v t$ where there are no constant. But even then, if the model is only approximate (speed is not really constant), it might be better to leave in a constant even if it cannot be interpreted.

There are also special models which leave out the intercept. One example is paired data, twin studies.

$\endgroup$
4
$\begingroup$

Full revision of my thoughts. Indeed dropping the intercept will cause a bias problem.

Have you considered centering your data so an intercept would have some meaning and avoid explaining how some (unreasonable) values could give negative values? If you adjust all three explanatory variables by subtract the mean sqrft, mean lotsize and mean bath, then the intercept will now indicate the value (of a house?) with average sdrft, lotsize, and baths.

This centering will not change the relative relationship of the independent variables. So, fitting the model on the centered data will still find baths as insignificant. Refit the model without the bath included. You may still get a large p-value for the intercept, but it should be included and you will have a model of the form y=a+b(sqrft)+c(lotsize).

$\endgroup$
2
$\begingroup$

I just spent some time answering a similar question posted by someone else, but it was closed. There are some great answers here, but the answer I provide is a bit simpler. It might be more suited to people who have a weak understanding of regression.

Q1: How do I interpret the intercept in my model?

In regression models, the goal is to minimise the amount of unexplained variance in an outcome variable:

y = b0 + b1⋅x + ϵ

where y is the predicted value of your outcome measure (e.g., log_blood_hg), b0 is the intercept, b1 is the slope, x is a predictor variable, and ϵ is residual error.

The intercept (b0) is the predicted mean value of y when all x = 0. In other words, it's the baseline value of y, before you've used any variables (e.g., species) to further minimise or explain the variance in log_blood_hg.

By adding a slope (which estimates how a one-unit increase/decrease in log_blood_hg changes with a one unit increase in x, e.g., species), we add to what we already know about the outcome variable, which is its baseline value (i.e. intercept), based on change in another variable.

Q2: When is it appropriate to include or not include the intercept, especially in regards to the fact that the models give very different results?

For simple models like this, it's never really appropriate to drop the intercept.

The models give different results when you drop the intercept because rather than grounding the slope in the baseline value of Y, it is forced to go through the origin of y, which is 0. Therefore, the slope gets steeper (i.e. more powerful and significant) because you've forced the line through the origin, not because it does a better job of minimizing the variance in y. In other words, you've artificially created a model which minimizes the variance in y by removing the intercept, or the initial grounding point for your model.

There are cases where removing the intercept is appropriate - such as when describing a phenomenon with a 0-intercept. You can read about that here, as well as more reasons why removing an intercept isn't a good idea.

$\endgroup$
1
$\begingroup$

This question has already many answers and basically they relate to the two cases (the first and second point below). But the answers make not clear why it is the case. While pondering about this and thinking about why the intercept would be different from other regressors (why should we never exclude the intercept term but is this not the case for other regressors?), I came to the idea that the intercept is not different. The intercept does have a special role like computations of the $R^2$ value and making the mean of residuals zero, but that does not make it that you can not remove it like you can with other regressor terms.

It can be ok to remove the intercept when ...

  1. When you have categorical variables. In that case the intercept is already in the column space of the regressors.

    Example this answer to 'Lm adding coefficients for different level of variable'.

  2. When you theoretically know that it should be zero.

    Example: fitting an exponential decay for which the function is $y(t) = y_0 e^{-ct}$.

    In some way you could see the $y_0$ as an intercept, because it behaves like an intercept when you plot the logarithm of $y$. But this function is different from $y(t) = y_0 e^{-ct} + b$ where this $b$ might be some baseline. Even when it is possible, due to measurement error, that the intercept/baseline $b$ is not zero, then one might whish to not include it (see next point).

  3. Not only when the intercept is theoretically exactly zero but also when the intercept is small.

    The variance in the estimate of the intercept might introduce an increased variance in the estimates or predictions. A model without the intercept could have better performance.

    In this way the intercept is like any other regressor variable that we might choose to eliminate based on some bad performance in a cross validation. If you do model selection and believe that regressor variables can be eliminated, the the same must be true for the intercept term.

    Because I am too lazy to prove this rigourously, I made an example in R code that demonstrates this

    set.seed(1)
    
    ### this function makes a linear fit
    ### either with or without intercept
    ### and it returns the squared error of the estimates 
    sim = function(intercept = TRUE){
    
      ### the data
      x = -10:10
      u = 0.1+x
      y = u+rnorm(length(x))
    
      ### fitting
      if (intercept == TRUE) {
        mod = lm(y ~ 1+x)
      } else {
        mod = lm(y ~ 0+x)
      }
    
      ### return the error in the estimates 
      sum((predict(mod)-u)^2)
    }
    
    ### perform simulations 
    r1 = replicate(10^4,sim(TRUE))
    r0 = replicate(10^4,sim(FALSE))
    
    ### with intercept 1.981789
    mean(r1)
    ### without intercept 1.206731
    mean(r0)
    

    Sidenote: Not including the intercept (or any other parameter), just to reduce the variance is a bit rigorous. Alternatives to not including the intercept could be some regularisation.

The reason why we say that we should never exclude the intercept is, I guess, not because there is some theoretical reason that should give the intercept some advantage that other regressor terms do not have. But instead, it is because in practice the intercept is non zero and large enough to be important for the estimates.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.