2
$\begingroup$

I ran an Ordinary Least Squares model and found the constant/intercept is more significant than all the other features. When the intercept is included, the $R^2$ is 45%. When I remove the intercept, the $R^2$ drops to 29%.

The intercept also has the lowest p-value compared to all the other features.

Moreover, I used StandardScaler to scale the features used.

Why would the intercept be so significant?

Example code:

model_2 = sm.OLS(df_reg_y.astype(float), sm.add_constant(X_scaled.astype(float))).fit()

This area circled in purple is the scatter plot of the target variable vs the most significant feature. The histogram below it is the distribution of the target variable.

enter image description here

Edit: I realized that I forgot to scale the target variable. The issue was fixed after I scaled the features and target variable together.

Improve this question
$\endgroup$
7
  • $\begingroup$ Hard to tell without data, your data may be partially constant, hence why it has such a big effect. $\endgroup$
    – user2974951
    Commented Oct 10, 2018 at 13:27
  • $\begingroup$ @user2974951 So how would I intercept this finding? All the other features are relatively insignificant compared to the intercept. The most significant feature only accounts for 6% of R-squared. $\endgroup$
    – SAKURA
    Commented Oct 10, 2018 at 13:37
  • $\begingroup$ Your data may not be linearly dependent, that is constant / non-changing. You should plot the results and add them here. $\endgroup$
    – user2974951
    Commented Oct 10, 2018 at 13:39
  • $\begingroup$ Did you scale the response variable too? $\endgroup$
    – Vincent Guillemot
    Commented Oct 10, 2018 at 13:46
  • $\begingroup$ @VincentGuillemot Thank you! I made a silly mistake by not scaling the target variable. Now the y-intercept does not change R-squared. I'd accept you as the answer $\endgroup$
    – SAKURA
    Commented Oct 10, 2018 at 14:04

2 Answers 2

Reset to default
1
$\begingroup$

Noting this in an answer as it solved the problem and the community has voted to keep the question open:

The intercept is significant because the mean value of the response is detectably greater than zero. As you discovered, if you rescale the response variable by subtracting the mean, the intercept will be zero and no longer significant in subsequent analyses. And the $R^2$ will be identical for models with and without the intercept term.

Improve this answer
$\endgroup$
0
$\begingroup$

The intercept is there to account for the mean value of your dependent variable. By setting the intercept to 0, you are biasing your estimator, i.e. making the mean of the error different from 0.

The R squared is sensitive to this bias change. If you want a measure for the goodness of fit that only considers variance (is insensitive to bias) and behaves like R squared, you can go for explained variance.

In other words, R squared is a summary statistic, but it does not tell you whether it is your bias or your variance that is restricting the goodness of your fit.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.