I am running linear regression models and wondering what the conditions are for removing the intercept term.
In comparing results from two different regressions where one has the intercept and the other does not, I notice that the $R^2$ of the function without the intercept is much higher. Are there certain conditions or assumptions I should be following to make sure the removal of the intercept term is valid?
The shortest answer: never, unless you are sure that your linear approximation of the data generating process (linear regression model) either by some theoretical or any other reasons is forced to go through the origin. If not the other regression parameters will be biased even if intercept is statistically insignificant (strange but it is so, consult Brooks Introductory Econometrics for instance). Finally, as I do often explain to my students, by leaving the intercept term you insure that the residual term is zero-mean.
For your two models case we need more context. It may happen that linear model is not suitable here. For example, you need to log transform first if the model is multiplicative. Having exponentially growing processes it may occasionally happen that $R^2$ for the model without the intercept is "much" higher.
Screen the data, test the model with RESET test or any other linear specification test, this may help to see if my guess is true. And, building the models highest $R^2$ is one of the last statistical properties I do really concern about, but it is nice to present to the people who are not so well familiar with econometrics (there are many dirty tricks to make determination close to 1 :)).
Removing the intercept is a different model, but there are plenty of examples where it is legitimate. Answers so far have already discussed in detail the example where the true intercept is 0. I will focus on a few examples where we may be interested in an atypical model parametrization.
Example 1: The ANOVA-style Model. For categorical variables, we typically create binary vectors encoding group membership. The standard regression model is parametrized as intercept + k - 1 dummy vectors. The intercept codes the expected value for the "reference" group, or the omitted vector, and the remaining vectors test the difference between each group and the reference. But in some cases, it may be useful to have each groups' expected value.
dat <- mtcars
dat$vs <- factor(dat$vs)
## intercept model: vs coefficient becomes difference
lm(mpg ~ vs + hp, data = dat)
Coefficients:
(Intercept) vs1 hp
26.96300 2.57622 -0.05453
## no intercept: two vs coefficients,
## conditional expectations for both groups
lm(mpg ~ 0 + vs + hp, data = dat)
Coefficients:
vs0 vs1 hp
26.96300 29.53922 -0.05453
Example 2: The case of standardized data. In some cases, one may be working with standardized data. In this case, the intercept is 0 by design. I think a classic example of this was old style structural equation models or factor, which operated just on the covariance matrices of data. In the case below, it is probably a good idea to estimate the intercept anyway, if only to drop the additional degree of freedom (which you really should have lost anyway because the mean was estimated), but there are a handful of situations where by construction, means may be 0 (e.g., certain experiments where participants assign ratings, but are constrained to give out equal positives and negatives).
dat <- as.data.frame(scale(mtcars))
## intercept is 0 by design
lm(mpg ~ hp + wt, data = dat)
Coefficients:
(Intercept) hp wt
3.813e-17 -3.615e-01 -6.296e-01
## leaving the intercept out
lm(mpg ~ 0 + hp + wt, data = dat)
Coefficients:
hp wt
-0.3615 -0.6296
Example 3: Multivariate Models and Hidden Intercepts. This example is similar to the first in many ways. In this case, the data has been stacked so that two different variables are now in one long vector. A second variable encodes information about whether the response vector, y, belongs to mpg or disp. In this case, to get the separate intercepts for each outcome, you suppress the overall intercept and include both dummy vectors for measure. This is a sort of multivariate analysis. It is not typically done using lm() because you have repeated measures and should probably allow for the nonindepence. However, there are some interesting cases where this is necessary. For example when trying to do a mediation analysis with random effects, to get the full variance covariance matrix, you need both models estimated simultaneously, which can be done by stacking the data and some clever use of dummy vectors.
## stack data for multivariate analysis
dat <- reshape(mtcars, varying = c(1, 3), v.names = "y",
timevar = "measure", times = c("mpg", "disp"),
direction = "long")
dat$measure <- factor(dat$measure)
## two regressions with intercepts only
lm(cbind(mpg, disp) ~ 1, data = mtcars)
Coefficients:
mpg disp
(Intercept) 20.09 230.72
## using the stacked data, measure is difference
## between outcome means
lm(y ~ measure, data = dat)
Coefficients:
(Intercept) measurempg
230.7 -210.6
## separate 'intercept' for each outcome
lm(y ~ 0 + measure, data = dat)
Coefficients:
measuredisp measurempg
230.72 20.09
I am not arguing that intercepts should generally be removed, but it is good to be flexible.
There are good answers here. Two small things:
You shouldn't drop the intercept, regardless of whether you are likely or not to ever see all the explanatory variables having values of zero.
If you remove the intercept then the other estimates all become biased. Even if the true value of the intercept is approximately zero (which is all you can conclude from your data), you are messing around with the slopes if you force it to be exactly zero.
UNLESS - you are measuring something with a very clear and obvious physical model that demands intercept be zero (eg you have height, width and length of a rectangular prism as explanatory variables and the response variable is volume with some measurement error). If your response variable is value of the house, you definitely need to leave the intercept in.
OK, so you've changed the question a LOT
You can leave out the intercept when you know it's 0. That's it. And no, you can't do it because it's not significantly different from 0, you have to know it's 0 or your residuals are biased. And, in that case it is 0 so it won't make any difference if you leave it out... therefore, never leave it out.
The finding you have with $R^2$ suggests the data are not linear. And, given that you had area as a predictor that particular one is probably definitely not linear. You could transform the predictor to fix that.
Most multiple regression models include a constant term (i.e., the intercept), since this ensures that the model will be unbiased--i.e., the mean of the residuals will be exactly zero. (The coefficients in a regression model are estimated by least squares--i.e., minimizing the mean squared error. Now, the mean squared error is equal to the variance of the errors plus the square of their mean: this is a mathematical identity. Changing the value of the constant in the model changes the mean of the errors but doesn't affect the variance. Hence, if the sum of squared errors is to be minimized, the constant must be chosen such that the mean of the errors is zero.)
In a simple regression model, the constant represents the Y-intercept of the regression line, in unstandardized form. In a multiple regression model, the constant represents the value that would be predicted for the dependent variable if all the independent variables were simultaneously equal to zero--a situation which may not physically or economically meaningful. If you are not particularly interested in what would happen if all the independent variables were simultaneously zero, then you normally leave the constant in the model regardless of its statistical significance. In addition to ensuring that the in-sample errors are unbiased, the presence of the constant allows the regression line to "seek its own level" and provide the best fit to data which may only be locally linear.
However, in rare cases you may wish to exclude the constant from the model. This is a model-fitting option in the regression procedure in any software package, and it is sometimes referred to as regression through the origin, or RTO for short. Usually, this will be done only if:
An example of case (1) would be a model in which all variables--dependent and independent--represented first differences of other time series. If you are regressing the first difference of Y on the first difference of X, you are directly predicting changes in Y as a linear function of changes in X, without reference to the current levels of the variables. In this case it might be reasonable (although not required) to assume that Y should be unchanged, on the average, whenever X is unchanged--i.e., that Y should not have an upward or downward trend in the absence of any change in the level of X.
An example of case (2) would be a situation in which you wish to use a full set of seasonal indicator variables--e.g., you are using quarterly data, and you wish to include variables Q1, Q2, Q3, and Q4 representing additive seasonal effects. Thus, Q1 might look like 1 0 0 0 1 0 0 0 ..., Q2 would look like 0 1 0 0 0 1 0 0 ..., and so on. You could not use all four of these and a constant in the same model, since Q1+Q2+Q3+Q4 = 1 1 1 1 1 1 1 1 . . . . , which is the same as a constant term. I.e., the five variables Q1, Q2, Q3, Q4, and CONSTANT are not linearly independent: any one of them can be expressed as a linear combination of the other four. A technical prerequisite for fitting a linear regression model is that the independent variables must be linearly independent; otherwise the least-squares coefficients cannot be determined uniquely, and we say the regression "fails."
A word of warning: R-squared and the F statistic do not have the same meaning in an RTO model as they do in an ordinary regression model, and they are not calculated in the same way by all software. See this article for some caveats. You should not try to compare R-squared between models that do and do not include a constant term, although it is OK to compare the standard error of the regression.
Note that the term "independent" is used in (at least) three different ways in regression jargon: any single variable may be called an independent variable if it is being used as a predictor, rather than as the predictee. A group of variables is linearly independent if no one of them can be expressed exactly as a linear combination of the others. A pair of variables is said to be statistically independent if they are not only linearly independent but also utterly uninformative with respect to each other. In a regression model, you want your dependent variable to be statistically dependent on the independent variables, which must be linearly (but not necessarily statistically) independent among themselves.
Short answer: (almost) NEVER. In the linear regression model $$ y = \alpha + \beta x + \epsilon $$, if you set $\alpha=0$, then you say that you KNOW that the expected value of $y$ given $x=0$ is zero. You almost never know that.
$R^2$ becomes higher without intercept, not because the model is better, but because the definition of $R^2$ used is another one! $R^2$ is an expression of a comparison of the estimated model with some standard model, expressed as reduction in sum of squares compared to sum of squares with the standard model. In the model with intercept, the comparison sum of squares is around the mean. Without intercept, it is around zero! The last one is usually much higher, so it easier to get a large reduction in sum of squares.
Conclusion: DO NOT LEAVE THE INTERCEPT OUT OF THE MODEL (unless you really, really know what you are doing).
Some exceptions: One exception is a regression representing a one-way ANOVA with dummies for ALL the factor levels (usually one is left out) (but that is only seemingly an exception, the constant vector 1 is in the column space of the model matrix $X$.)
Compositional data compositional-data as predictors (but also only seemingly an exception) as an example see How to measure ML model performance across race via area level demographic info
Otherwise, such as physical relationships $s=v t$ where there are no constant. But even then, if the model is only approximate (speed is not really constant), it might be better to leave in a constant even if it cannot be interpreted.
There are also special models which leave out the intercept. One example is paired data, twin studies.
Full revision of my thoughts. Indeed dropping the intercept will cause a bias problem.
Have you considered centering your data so an intercept would have some meaning and avoid explaining how some (unreasonable) values could give negative values? If you adjust all three explanatory variables by subtract the mean sqrft, mean lotsize and mean bath, then the intercept will now indicate the value (of a house?) with average sdrft, lotsize, and baths.
This centering will not change the relative relationship of the independent variables. So, fitting the model on the centered data will still find baths as insignificant. Refit the model without the bath included. You may still get a large p-value for the intercept, but it should be included and you will have a model of the form y=a+b(sqrft)+c(lotsize).
I just spent some time answering a similar question posted by someone else, but it was closed. There are some great answers here, but the answer I provide is a bit simpler. It might be more suited to people who have a weak understanding of regression.
In regression models, the goal is to minimise the amount of unexplained variance in an outcome variable:
y = b0 + b1⋅x + ϵ
where y is the predicted value of your outcome measure (e.g., log_blood_hg), b0 is the intercept, b1 is the slope, x is a predictor variable, and ϵ is residual error.
The intercept (b0) is the predicted mean value of y when all x = 0. In other words, it's the baseline value of y, before you've used any variables (e.g., species) to further minimise or explain the variance in log_blood_hg.
By adding a slope (which estimates how a one-unit increase/decrease in log_blood_hg changes with a one unit increase in x, e.g., species), we add to what we already know about the outcome variable, which is its baseline value (i.e. intercept), based on change in another variable.
For simple models like this, it's never really appropriate to drop the intercept.
The models give different results when you drop the intercept because rather than grounding the slope in the baseline value of Y, it is forced to go through the origin of y, which is 0. Therefore, the slope gets steeper (i.e. more powerful and significant) because you've forced the line through the origin, not because it does a better job of minimizing the variance in y. In other words, you've artificially created a model which minimizes the variance in y by removing the intercept, or the initial grounding point for your model.
There are cases where removing the intercept is appropriate - such as when describing a phenomenon with a 0-intercept. You can read about that here, as well as more reasons why removing an intercept isn't a good idea.
This question has already many answers and basically they relate to the two cases (the first and second point below). But the answers make not clear why it is the case. While pondering about this and thinking about why the intercept would be different from other regressors (why should we never exclude the intercept term but is this not the case for other regressors?), I came to the idea that the intercept is not different. The intercept does have a special role like computations of the $R^2$ value and making the mean of residuals zero, but that does not make it that you can not remove it like you can with other regressor terms.
It can be ok to remove the intercept when ...
When you have categorical variables. In that case the intercept is already in the column space of the regressors.
Example this answer to 'Lm adding coefficients for different level of variable'.
When you theoretically know that it should be zero.
Example: fitting an exponential decay for which the function is $y(t) = y_0 e^{-ct}$.
In some way you could see the $y_0$ as an intercept, because it behaves like an intercept when you plot the logarithm of $y$. But this function is different from $y(t) = y_0 e^{-ct} + b$ where this $b$ might be some baseline. Even when it is possible, due to measurement error, that the intercept/baseline $b$ is not zero, then one might whish to not include it (see next point).
Not only when the intercept is theoretically exactly zero but also when the intercept is small.
The variance in the estimate of the intercept might introduce an increased variance in the estimates or predictions. A model without the intercept could have better performance.
In this way the intercept is like any other regressor variable that we might choose to eliminate based on some bad performance in a cross validation. If you do model selection and believe that regressor variables can be eliminated, the the same must be true for the intercept term.
Because I am too lazy to prove this rigourously, I made an example in R code that demonstrates this
set.seed(1)
### this function makes a linear fit
### either with or without intercept
### and it returns the squared error of the estimates
sim = function(intercept = TRUE){
### the data
x = -10:10
u = 0.1+x
y = u+rnorm(length(x))
### fitting
if (intercept == TRUE) {
mod = lm(y ~ 1+x)
} else {
mod = lm(y ~ 0+x)
}
### return the error in the estimates
sum((predict(mod)-u)^2)
}
### perform simulations
r1 = replicate(10^4,sim(TRUE))
r0 = replicate(10^4,sim(FALSE))
### with intercept 1.981789
mean(r1)
### without intercept 1.206731
mean(r0)
Sidenote: Not including the intercept (or any other parameter), just to reduce the variance is a bit rigorous. Alternatives to not including the intercept could be some regularisation.
The reason why we say that we should never exclude the intercept is, I guess, not because there is some theoretical reason that should give the intercept some advantage that other regressor terms do not have. But instead, it is because in practice the intercept is non zero and large enough to be important for the estimates.
When a categorical (group) explanatory variable is included, it is only a matter of model parametrization. Include the intercept if you want to compare other groups to a reference group. Exclude intercept if you want your coefficient estimates to correspond to adjusted group means and not to group differences.
If you have only continuous explanatory variables, exclude intercept only in those rare circumstances when you need to make sure that all explanatory variables being 0 produces model prediction 0.
