Many textbooks and papers said that intercept should not be suppressed. Recently, I used a training dataset to build a linear regression model with or without an intercept. I was surprised to find that the model without an intercept predicts better than that with an intercept in terms of rmse in an independent validation dataset. Is the prediction accuracy one of the reasons that I should use zero-intercept models?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ How big were the training and validation sample sizes? Maybe the model without an intercept was better just by chance. $\endgroup$– mark999Jan 26, 2012 at 4:15
-
$\begingroup$ The training sample size was 289 whereas the validation sample size was 406. By the way, how to determine the best training and validation sample sizes? $\endgroup$– KuJJan 26, 2012 at 5:22
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
5
Look carefully at how the rmse or other statistic is computed when comparing no-intercept models to intercept models. Sometimes the assumptions and calculations are different between the 2 models and one may fit worse, but look better because it is being divided by something much larger.
Without a reproducible example it is difficult to tell what may be contributing.
answered Jan 26, 2012 at 18:46
-
$\begingroup$ Rmse was calculated according to the formula (used to compare differences between two things that may vary, neither of which is accepted as the "standard") given in: en.wikipedia.org/wiki/Root-mean-square_deviation So the assumptions and calculations are the same between the 2 model-derived estimators. $\endgroup$– KuJJan 27, 2012 at 2:38
-
1$\begingroup$ How similar are your training and validation sets? You might consider doing 10-fold cross validation: split the data into 10 equal (or as equal as you can get) pieces randomly, then use 9 of them to train a model and the 10th as the validation piece, then repeate with each of the other 9 pieces being the validation set. Then repeate that whole process (starting with a new random split) 10 times or so. $\endgroup$ Jan 27, 2012 at 15:49
-
$\begingroup$ Y variable and X variables were different (P=0.01) between the training and validation sets. However, rmse was still lower with the model with no intercept when I used the R package MatchIt to match the training and validation sets. In contrast, rmse became similar when I used the DAAG package (cv.lm) to 10-fold cross-validate the combined dataset. Does this mean that 10-fold cross-validation is better than a simple training set and a validation set? $\endgroup$– KuJJan 28, 2012 at 10:03
-
2$\begingroup$ Yes the cross-validation tends to be better. Look to see if there are other ways that your model is over specified, it is very unusual for a non-intercept model to fit better. $\endgroup$ Jan 28, 2012 at 23:42
-
1$\begingroup$ In "To Explain or to Predict?" projecteuclid.org/… Professor Galit Shmueli said that sometimes a less true model can predict better than a truer model. I think this may be one of the reasons of this case. $\endgroup$– KuJFeb 3, 2012 at 15:02
$\begingroup$
$\endgroup$
2
I don't think you should choose models simply because they work better in a particular sample, although it is good that you used a training and validation sample.
Rather, look at what the models say about your situation. In some cases a zero-intercept model makes sense. If the DV ought to be 0 when all the IVs are 0, then use a zero-intercept model. Otherwise, don't.
Substantive knowledge should guide statistics, not the other way around
answered Jan 26, 2012 at 11:08
-
2$\begingroup$ The reason given in your second paragraph, while intuitive, is often not a strong enough one to suppress the intercept in many such situations. This point is addressed more fully in a couple other questions on this site. $\endgroup$– cardinalJan 26, 2012 at 13:51
-
4$\begingroup$ In method (or instrument) comparison studies (e.g. the comparison of oximeter A and oximeter B), the DV (oxygen level) ought to be 0 when all the IVs (oxygen levels) are 0. However, the intercept should not be ignored if I want to calibrate (or exchange) oximeter A with oximter B. $\endgroup$– KuJJan 26, 2012 at 14:28
$\begingroup$
$\endgroup$
0
A no intercept model may make sense if two conditions are met. First, there should be a reasonable subject matter knowledge expectation for the intercept to be zero. Second, there should be a reasonable subject matter knowledge expection for the regression line to remain a straight line as you approach zero. Even if both conditions are satisfied, it is wise to run an analysis with an intercept term and verify that the intercept is not significantly different from zero.
(I am assuming that you are talking about a continuous Y and a continuous X.)
$\begingroup$
$\endgroup$
3
This would be understandable if the intercept you obtained was merely noise --not sig. different from zero. (Am I right that the standardized regression coefficients were nearly the same in both models?) If so I don't think you should generalize from this example. When intercepts are sig. and substantial, they add something meaningful to predictive accuracy.
-
$\begingroup$ 1. The standardized regression coefficients were not the same (0.91 and 1.02) for the model with and without an intercept). 2. The intercept was 9.5 (s.e. 1.7, p<0.001). 3. As far as I know, many papers suggest not to suppress the intercept even if the intercept was not significant from zero. $\endgroup$– KuJJan 26, 2012 at 2:51
-
$\begingroup$ If there are dummy variables in the regression, doesn't the intercept represent the value if all dummies are 0-coded for that observation? Not sure if this applies here. $\endgroup$– MichelleJan 26, 2012 at 9:07
-
$\begingroup$
$\endgroup$
3
In linear regression, you are fitting:
$y = f(\beta, X) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots $
You fit $\beta$ given training data $(X, Y)$ Suppose you drop the $\beta_0$ and fit the model, will the error in the fit:
$ \sum_i (y_i- f(\beta, X_i) )^2$
be larger than if you included it? In all (non-degenerate) cases you can prove the error will be the same or lower (on the training data) when you include $\beta_0$ since the model is free to use this parameter to reduce the error if it is present and helps, and will set it to zero if it doesn't help. Further, suppose you added a large constant to y (assume your output needed to be $+10000$ than in your original training data), and refit the model, then $\beta_0$ clearly becomes very important.
Perhaps you're referring to regularized models when you say "suppressed". The L1 and L2 regularized, these methods prefer to keep coefficients close to zero (and you should have already mean and variance normalized your $X$ beforehand to make this step a sensible one. In regularization, you then have a choice whether to include the intercept term (should we prefer also to have a small $\beta_0$?). Again, in most cases (all cases?), you're better off not regularizing $\beta_0$, since its unlikely to reduce overfitting and shrinks the space of representable functions (by excluding those with high $\beta_0$) leading to higher error.
Side note: scikit's logistic regression regularizes the intercept by default. Anyone know why: http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html ? I don't think its a good idea.
-
$\begingroup$ Re sklearn regularizing the intercept: that's only the case if using the liblinear solver and they explicitly note it's bad (scikit-learn.org/stable/modules/…). I guess the intercept_scaling parameter is included in LogisticRegression to mitigate this. $\endgroup$– erobertcApr 17, 2020 at 21:11
-
$\begingroup$ Scaling doesn't help if you plan on interpreting the parameters as odds-ratios. $\endgroup$ Apr 17, 2020 at 21:12
-
$\begingroup$ I'm glad they updated the docs. Many hours lost here. $\endgroup$ Apr 17, 2020 at 21:13