4
$\begingroup$

I'm working in Python with statsmodels. I have a response variable y and a design matrix X from which I have already removed the most strongly correlated (redundant) predictors. I add a constant and check the VIFs:

> from statsmodels.stats.outliers_influence import variance_inflation_factor as vif
> Xc = add_constant(X)
> vifs = [vif(Xc.values, i) for i in range(len(Xc.columns))]
> pd.Series(data=vifs, index=Xc.columns).sort_values(ascending=False)

const   251.828124
x1      4.007442
x2      3.768146
x3      3.151422
x4      3.093936
x5      2.252909
x6      2.182324
x7      2.121366
x8      2.095420
x9      2.026337
x10     2.015635
x11     1.732534
x12     1.514766
dtype: float64

The constant has a very high VIF but all the other values seem reasonable. I estimate a regression model:

> import statsmodels.api as sm
> model = sm.OLS(y, Xc)
> result = model.fit()
> print(result.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.219
Model:                            OLS   Adj. R-squared:                  0.218
Method:                 Least Squares   F-statistic:                     277.5
Date:                Thu, 08 Mar 2018   Prob (F-statistic):               0.00
Time:                        09:36:09   Log-Likelihood:                -15948.
No. Observations:               11915   AIC:                         3.192e+04
Df Residuals:                   11902   BIC:                         3.202e+04
Df Model:                          12                                         
Covariance Type:            nonrobust                                         
===============================================================================================
                          coef    std err          t      P>|t|      [0.025      0.975]
-----------------------------------------------------------------------------------------------
const                  -1.2364      0.134     -9.212      0.000      -1.499      -0.973
x1                      0.0066      0.001      4.486      0.000       0.004       0.009
x2                     -0.0654      0.030     -2.214      0.027      -0.123      -0.007
x3                     -0.1051      0.032     -3.294      0.001      -0.168      -0.043
x4                     -0.0135      0.002     -8.922      0.000      -0.016      -0.011
x5                      0.0048      0.001      8.383      0.000       0.004       0.006
x6                      0.0276      0.013      2.049      0.041       0.001       0.054
x7                      0.5270      0.040     13.029      0.000       0.448       0.606
x8                      0.0113      0.001      8.975      0.000       0.009       0.014
x9                      0.0013      0.000      3.181      0.001       0.000       0.002
x10                     0.0067      0.001     10.322      0.000       0.005       0.008
x11                    -0.0051      0.001     -4.133      0.000      -0.007      -0.003
x12                    -0.1398      0.017     -8.243      0.000      -0.173      -0.107
==============================================================================
Omnibus:              311.108   Durbin-Watson:                   1.688
Prob(Omnibus):          0.000   Jarque-Bera (JB):              376.035
Skew:                   0.341   Prob(JB):                     2.21e-82
Kurtosis:               3.541   Cond. No.                     2.08e+03
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 2.08e+03. This might indicate that there are
strong multicollinearity or other numerical problems.

My coefficients are signed in the right direction and are all significant at the .05 level. However, the condition number is extremely high, suggesting severe multicollinearity. If I remove the constant and re-estimate, the condition number drops to 642 and the multicollinearity warning disappears. Moreover if I standardize y and X and re-estimate a model:

> from scipy.stats.mstats import zscore
> y_stdrd = pd.Series(zscore(y), index=y.index, name=y.name)
> X_stdrd = pd.DataFrame(data=zscore(X), index=X.index, columns=X.columns)
> Xc_stdrd = add_constant(X_stdrd)
> model_stdrd = sm.OLS(y_stdrd, Xc_stdrd)
> result_stdrd = model_stdrd.fit()
> print(result_stdrd.summary())

                                OLS Regression Results                            
==============================================================================
Dep. Variable:                      y   R-squared:                       0.219
Model:                            OLS   Adj. R-squared:                  0.218
Method:                 Least Squares   F-statistic:                     277.5
Date:                Thu, 08 Mar 2018   Prob (F-statistic):               0.00
Time:                        09:42:39   Log-Likelihood:                -15437.
No. Observations:               11915   AIC:                         3.090e+04
Df Residuals:                   11902   BIC:                         3.100e+04
Df Model:                          12                                         
Covariance Type:            nonrobust                                         
===============================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
-----------------------------------------------------------------------------------------------
const             6.462e-17      0.008   7.98e-15      1.000      -0.016       0.016
x1                   0.0478      0.011      4.486      0.000       0.027       0.069
x1                  -0.0261      0.012     -2.214      0.027      -0.049      -0.003
x2                  -0.0386      0.012     -3.294      0.001      -0.062      -0.016
x3                  -0.1026      0.012     -8.922      0.000      -0.125      -0.080
x4                   0.1360      0.016      8.383      0.000       0.104       0.168
x5                   0.0295      0.014      2.049      0.041       0.001       0.058
x6                   0.1857      0.014     13.029      0.000       0.158       0.214
x7                   0.1074      0.012      8.975      0.000       0.084       0.131
x8                   0.0317      0.010      3.181      0.001       0.012       0.051
x9                   0.1624      0.016     10.322      0.000       0.132       0.193
x10                 -0.0477      0.012     -4.133      0.000      -0.070      -0.025
x11                 -0.1002      0.012     -8.243      0.000      -0.124      -0.076
==============================================================================
Omnibus:            311.108   Durbin-Watson:                   1.688
Prob(Omnibus):        0.000   Jarque-Bera (JB):              376.035
Skew:                 0.341   Prob(JB):                     2.21e-82
Kurtosis:             3.541   Cond. No.                         4.64
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Now the condition number is a reasonable 4.64 and the severe multicollinearity has disappeared. However my constant is no longer statistically significant.

Questions:

  1. What does it mean for a constant to have such a high VIF (> 250) while all other predictors' VIFs are around 4 or below?

  2. Related to #1, is the constant's VIF the reason why my unstandardized model reports the high condition number (> 2000) and warns about strong multicollinearity?

  3. Why does standardization fix this high condition number?

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
6
$\begingroup$

Ok, combining several comments and some further Googling, I think I have the answer. This is a scaling problem: for a very simplified example see here.

statsmodels reports the condition number of the design matrix and not of a standardized design matrix. It calculates this as the ratio of the largest eigenvalue in the design matrix to the smallest. In other words, the large condition number in this case results from scaling rather than from multicollinearity. If we have just one variable with units in the thousands (ie, a large eigenvalue) and add a constant with units of 1 (ie, a small eigenvalue), we'll get a large condition number as the ratio, and statsmodels warns of multicollinearity.

statsmodels sets its threshold for warning about multicollinearity at 30, which is pretty sensitive for real-world applied analysis and an unstandardized design matrix given this scaling issue.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The threshold for the condition number warning has been increased to 1000, which is still low if the large condition number comes mainly from scaling problems. $\endgroup$
    – Josef
    Mar 10, 2018 at 14:35
  • $\begingroup$ It explains a difference of condition number obtained from statsmodels and the collin command of Stata. $\endgroup$
    – T_T
    Jul 20, 2021 at 3:17
1
$\begingroup$

The variance inflation factor for constant should be high (actually infinite) since it is $\frac{1}{1-R^2_i}$ where the $R^2$ is from the regression

$$X_{i} = \alpha + \vec{\beta}^\top \vec{X}_{(-i)}$$

where $\vec{X}_{(-i)}$ are all the covariates but $i$. The intercept $\alpha$ should perfectly explain a constant. Thus, you should not look at the VIF for the constant. Do notice that you still have to add this to the design matrix when you call the vif function that you use. The post I linked to also explain why the VIF for the constant is not infinite (there is not constant in the regression for the $R^2_i$ for the intercept).

The above does not explain the condition number in your regression.

First answer

What does it mean for a constant to have such a high VIF (> 250) while all other predictors' VIFs are around 4 or below?

How have you coded your dummy variables? You may have issues with your dummy coding.

Why does standardization fix this high condition number and also make my constant insignificant?

Because the constant becomes zero when you standardize (assuming that the software you use uses $0/0 = 0$ or you get $0/\epsilon$ where $\epsilon$ is small).

Improve this answer
$\endgroup$
4
  • $\begingroup$ There are two race dummies. dummy_black is coded 1 if the tract is majority black and 0 otherwise. dummy_hispanic is coded 1 if the tract is majority hispanic and 0 otherwise. The white/other dummy is accordingly left out of the analysis. $\endgroup$
    – eos
    Mar 8, 2018 at 21:16
  • $\begingroup$ Do you have two or more variables where you can form a linear combination which almost is constant? $\endgroup$
    – Benjamin Christoffersen
    Mar 8, 2018 at 21:48
  • $\begingroup$ How would I ascertain that across the entire feature set? $\endgroup$
    – eos
    Mar 8, 2018 at 22:04
  • 1
    $\begingroup$ A google search on "QR pivoting to detect rank deficient in Python" lead me to this post. Hope it helps. $\endgroup$
    – Benjamin Christoffersen
    Mar 8, 2018 at 22:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.