Many textbooks and papers said that intercept should not be suppressed. Recently, I used a training dataset to build a linear regression model with or without an intercept. I was surprised to find that the model without an intercept predicts better than that with an intercept in terms of rmse in an independent validation dataset. Is the prediction accuracy one of the reasons that I should use zero-intercept models?

  • $\begingroup$ How big were the training and validation sample sizes? Maybe the model without an intercept was better just by chance. $\endgroup$ – mark999 Jan 26 '12 at 4:15
  • $\begingroup$ The training sample size was 289 whereas the validation sample size was 406. By the way, how to determine the best training and validation sample sizes? $\endgroup$ – KuJ Jan 26 '12 at 5:22

Look carefully at how the rmse or other statistic is computed when comparing no-intercept models to intercept models. Sometimes the assumptions and calculations are different between the 2 models and one may fit worse, but look better because it is being divided by something much larger.

Without a reproducible example it is difficult to tell what may be contributing.

  • $\begingroup$ Rmse was calculated according to the formula (used to compare differences between two things that may vary, neither of which is accepted as the "standard") given in: en.wikipedia.org/wiki/Root-mean-square_deviation So the assumptions and calculations are the same between the 2 model-derived estimators. $\endgroup$ – KuJ Jan 27 '12 at 2:38
  • 1
    $\begingroup$ How similar are your training and validation sets? You might consider doing 10-fold cross validation: split the data into 10 equal (or as equal as you can get) pieces randomly, then use 9 of them to train a model and the 10th as the validation piece, then repeate with each of the other 9 pieces being the validation set. Then repeate that whole process (starting with a new random split) 10 times or so. $\endgroup$ – Greg Snow Jan 27 '12 at 15:49
  • $\begingroup$ Y variable and X variables were different (P=0.01) between the training and validation sets. However, rmse was still lower with the model with no intercept when I used the R package MatchIt to match the training and validation sets. In contrast, rmse became similar when I used the DAAG package (cv.lm) to 10-fold cross-validate the combined dataset. Does this mean that 10-fold cross-validation is better than a simple training set and a validation set? $\endgroup$ – KuJ Jan 28 '12 at 10:03
  • 2
    $\begingroup$ Yes the cross-validation tends to be better. Look to see if there are other ways that your model is over specified, it is very unusual for a non-intercept model to fit better. $\endgroup$ – Greg Snow Jan 28 '12 at 23:42
  • 1
    $\begingroup$ In "To Explain or to Predict?" projecteuclid.org/… Professor Galit Shmueli said that sometimes a less true model can predict better than a truer model. I think this may be one of the reasons of this case. $\endgroup$ – KuJ Feb 3 '12 at 15:02

I don't think you should choose models simply because they work better in a particular sample, although it is good that you used a training and validation sample.

Rather, look at what the models say about your situation. In some cases a zero-intercept model makes sense. If the DV ought to be 0 when all the IVs are 0, then use a zero-intercept model. Otherwise, don't.

Substantive knowledge should guide statistics, not the other way around

  • 2
    $\begingroup$ The reason given in your second paragraph, while intuitive, is often not a strong enough one to suppress the intercept in many such situations. This point is addressed more fully in a couple other questions on this site. $\endgroup$ – cardinal Jan 26 '12 at 13:51
  • 4
    $\begingroup$ In method (or instrument) comparison studies (e.g. the comparison of oximeter A and oximeter B), the DV (oxygen level) ought to be 0 when all the IVs (oxygen levels) are 0. However, the intercept should not be ignored if I want to calibrate (or exchange) oximeter A with oximter B. $\endgroup$ – KuJ Jan 26 '12 at 14:28

A no intercept model may make sense if two conditions are met. First, there should be a reasonable subject matter knowledge expectation for the intercept to be zero. Second, there should be a reasonable subject matter knowledge expection for the regression line to remain a straight line as you approach zero. Even if both conditions are satisfied, it is wise to run an analysis with an intercept term and verify that the intercept is not significantly different from zero.

(I am assuming that you are talking about a continuous Y and a continuous X.)


This would be understandable if the intercept you obtained was merely noise --not sig. different from zero. (Am I right that the standardized regression coefficients were nearly the same in both models?) If so I don't think you should generalize from this example. When intercepts are sig. and substantial, they add something meaningful to predictive accuracy.

  • $\begingroup$ 1. The standardized regression coefficients were not the same (0.91 and 1.02) for the model with and without an intercept). 2. The intercept was 9.5 (s.e. 1.7, p<0.001). 3. As far as I know, many papers suggest not to suppress the intercept even if the intercept was not significant from zero. $\endgroup$ – KuJ Jan 26 '12 at 2:51
  • $\begingroup$ If there are dummy variables in the regression, doesn't the intercept represent the value if all dummies are 0-coded for that observation? Not sure if this applies here. $\endgroup$ – Michelle Jan 26 '12 at 9:07
  • $\begingroup$ No, there were no dummy variables. $\endgroup$ – KuJ Jan 26 '12 at 14:00

In linear regression, you are fitting:

$y = f(\beta, X) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots $

You fit $\beta$ given training data $(X, Y)$ Suppose you drop the $\beta_0$ and fit the model, will the error in the fit:

$ \sum_i (y_i- f(\beta, X_i) )^2$

be larger than if you included it? In all (non-degenerate) cases you can prove the error will be the same or lower (on the training data) when you include $\beta_0$ since the model is free to use this parameter to reduce the error if it is present and helps, and will set it to zero if it doesn't help. Further, suppose you added a large constant to y (assume your output needed to be $+10000$ than in your original training data), and refit the model, then $\beta_0$ clearly becomes very important.

Perhaps you're referring to regularized models when you say "suppressed". The L1 and L2 regularized, these methods prefer to keep coefficients close to zero (and you should have already mean and variance normalized your $X$ beforehand to make this step a sensible one. In regularization, you then have a choice whether to include the intercept term (should we prefer also to have a small $\beta_0$?). Again, in most cases (all cases?), you're better off not regularizing $\beta_0$, since its unlikely to reduce overfitting and shrinks the space of representable functions (by excluding those with high $\beta_0$) leading to higher error.

Side note: scikit's logistic regression regularizes the intercept by default. Anyone know why: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html ? I don't think its a good idea.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.