I'm working in Python with statsmodels. I have a response variable y
and a design matrix X
from which I have already removed the most strongly correlated (redundant) predictors. I add a constant and check the VIFs:
> from statsmodels.stats.outliers_influence import variance_inflation_factor as vif
> Xc = add_constant(X)
> vifs = [vif(Xc.values, i) for i in range(len(Xc.columns))]
> pd.Series(data=vifs, index=Xc.columns).sort_values(ascending=False)
const 251.828124
x1 4.007442
x2 3.768146
x3 3.151422
x4 3.093936
x5 2.252909
x6 2.182324
x7 2.121366
x8 2.095420
x9 2.026337
x10 2.015635
x11 1.732534
x12 1.514766
dtype: float64
The constant has a very high VIF but all the other values seem reasonable. I estimate a regression model:
> import statsmodels.api as sm
> model = sm.OLS(y, Xc)
> result = model.fit()
> print(result.summary())
OLS Regression Results
==============================================================================
Dep. Variable: y R-squared: 0.219
Model: OLS Adj. R-squared: 0.218
Method: Least Squares F-statistic: 277.5
Date: Thu, 08 Mar 2018 Prob (F-statistic): 0.00
Time: 09:36:09 Log-Likelihood: -15948.
No. Observations: 11915 AIC: 3.192e+04
Df Residuals: 11902 BIC: 3.202e+04
Df Model: 12
Covariance Type: nonrobust
===============================================================================================
coef std err t P>|t| [0.025 0.975]
-----------------------------------------------------------------------------------------------
const -1.2364 0.134 -9.212 0.000 -1.499 -0.973
x1 0.0066 0.001 4.486 0.000 0.004 0.009
x2 -0.0654 0.030 -2.214 0.027 -0.123 -0.007
x3 -0.1051 0.032 -3.294 0.001 -0.168 -0.043
x4 -0.0135 0.002 -8.922 0.000 -0.016 -0.011
x5 0.0048 0.001 8.383 0.000 0.004 0.006
x6 0.0276 0.013 2.049 0.041 0.001 0.054
x7 0.5270 0.040 13.029 0.000 0.448 0.606
x8 0.0113 0.001 8.975 0.000 0.009 0.014
x9 0.0013 0.000 3.181 0.001 0.000 0.002
x10 0.0067 0.001 10.322 0.000 0.005 0.008
x11 -0.0051 0.001 -4.133 0.000 -0.007 -0.003
x12 -0.1398 0.017 -8.243 0.000 -0.173 -0.107
==============================================================================
Omnibus: 311.108 Durbin-Watson: 1.688
Prob(Omnibus): 0.000 Jarque-Bera (JB): 376.035
Skew: 0.341 Prob(JB): 2.21e-82
Kurtosis: 3.541 Cond. No. 2.08e+03
==============================================================================
Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.08e+03. This might indicate that there are
strong multicollinearity or other numerical problems.
My coefficients are signed in the right direction and are all significant at the .05 level. However, the condition number is extremely high, suggesting severe multicollinearity. If I remove the constant and re-estimate, the condition number drops to 642 and the multicollinearity warning disappears. Moreover if I standardize y
and X
and re-estimate a model:
> from scipy.stats.mstats import zscore
> y_stdrd = pd.Series(zscore(y), index=y.index, name=y.name)
> X_stdrd = pd.DataFrame(data=zscore(X), index=X.index, columns=X.columns)
> Xc_stdrd = add_constant(X_stdrd)
> model_stdrd = sm.OLS(y_stdrd, Xc_stdrd)
> result_stdrd = model_stdrd.fit()
> print(result_stdrd.summary())
OLS Regression Results
==============================================================================
Dep. Variable: y R-squared: 0.219
Model: OLS Adj. R-squared: 0.218
Method: Least Squares F-statistic: 277.5
Date: Thu, 08 Mar 2018 Prob (F-statistic): 0.00
Time: 09:42:39 Log-Likelihood: -15437.
No. Observations: 11915 AIC: 3.090e+04
Df Residuals: 11902 BIC: 3.100e+04
Df Model: 12
Covariance Type: nonrobust
===============================================================================================
coef std err t P>|t| [0.025 0.975]
-----------------------------------------------------------------------------------------------
const 6.462e-17 0.008 7.98e-15 1.000 -0.016 0.016
x1 0.0478 0.011 4.486 0.000 0.027 0.069
x1 -0.0261 0.012 -2.214 0.027 -0.049 -0.003
x2 -0.0386 0.012 -3.294 0.001 -0.062 -0.016
x3 -0.1026 0.012 -8.922 0.000 -0.125 -0.080
x4 0.1360 0.016 8.383 0.000 0.104 0.168
x5 0.0295 0.014 2.049 0.041 0.001 0.058
x6 0.1857 0.014 13.029 0.000 0.158 0.214
x7 0.1074 0.012 8.975 0.000 0.084 0.131
x8 0.0317 0.010 3.181 0.001 0.012 0.051
x9 0.1624 0.016 10.322 0.000 0.132 0.193
x10 -0.0477 0.012 -4.133 0.000 -0.070 -0.025
x11 -0.1002 0.012 -8.243 0.000 -0.124 -0.076
==============================================================================
Omnibus: 311.108 Durbin-Watson: 1.688
Prob(Omnibus): 0.000 Jarque-Bera (JB): 376.035
Skew: 0.341 Prob(JB): 2.21e-82
Kurtosis: 3.541 Cond. No. 4.64
==============================================================================
Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
Now the condition number is a reasonable 4.64 and the severe multicollinearity has disappeared. However my constant is no longer statistically significant.
Questions:
What does it mean for a constant to have such a high VIF (> 250) while all other predictors' VIFs are around 4 or below?
Related to #1, is the constant's VIF the reason why my unstandardized model reports the high condition number (> 2000) and warns about strong multicollinearity?
Why does standardization fix this high condition number?