12
$\begingroup$

I'm pretty new to regression analysis, and I'm using python's statsmodels to look at the relationship between GDP/health/social services spending and health outcomes (DALYs) across the OECD. Just to give an idea of the data I'm using, this is a scatter matrix with the diagonal being the kernel density estimate:

Scatter matrix results

When I run a simple regression of dalyrate on social_exp, the result shows a warning about a high condition number:

import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
import statsmodels.api as sm

poly_1 = smf.ols(formula='dalyrate ~ 1 + social_exp', data=model_df, missing='drop').fit()
print poly_1.summary()

                            OLS Regression Results                            
==============================================================================
Dep. Variable:               dalyrate   R-squared:                       0.253
Model:                            OLS   Adj. R-squared:                  0.248
Method:                 Least Squares   F-statistic:                     46.85
Date:                Fri, 28 Oct 2016   Prob (F-statistic):           2.30e-10
Time:                        12:56:43   Log-Likelihood:                -1336.8
No. Observations:                 140   AIC:                             2678.
Df Residuals:                     138   BIC:                             2683.
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept   2.705e+04    635.828     42.541      0.000      2.58e+04  2.83e+04
social_exp    -0.9303      0.136     -6.845      0.000        -1.199    -0.662
==============================================================================
Omnibus:                       34.504   Durbin-Watson:                   0.907
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               78.046
Skew:                           1.017   Prob(JB):                     1.13e-17
Kurtosis:                       6.039   Cond. No.                     1.03e+04
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.03e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

From what I've read, multicollinearity shouldn't be an issue with a single variable regression, so what could be causing this problem? It seems like both the gdp_cap and dalyrate variables are positively skewed in the scatter matrix, so do I need to normalize or standardize the data in order to prevent this warning? It would be ideal if I didn't need to because the units on the coefficient are helpful for interpreting the results, but obviously I'd do so if necessary.

I also get a similar warning with a multivariate regression:

poly_2 = smf.ols(formula='dalyrate ~ 1 + social_exp + health_exp + gdp_cap', data=model_df, missing='drop').fit()
print poly_2.summary()


                            OLS Regression Results                            
==============================================================================
Dep. Variable:               dalyrate   R-squared:                       0.406
Model:                            OLS   Adj. R-squared:                  0.393
Method:                 Least Squares   F-statistic:                     30.98
Date:                Fri, 28 Oct 2016   Prob (F-statistic):           2.51e-15
Time:                        13:04:35   Log-Likelihood:                -1320.8
No. Observations:                 140   AIC:                             2650.
Df Residuals:                     136   BIC:                             2661.
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept    2.86e+04    631.270     45.308      0.000      2.74e+04  2.98e+04
social_exp     0.0379      0.208      0.182      0.855        -0.373     0.449
health_exp    -1.1555      0.889     -1.300      0.196        -2.914     0.603
gdp_cap       -0.1412      0.053     -2.662      0.009        -0.246    -0.036
==============================================================================
Omnibus:                       53.794   Durbin-Watson:                   0.663
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              157.258
Skew:                           1.490   Prob(JB):                     7.11e-35
Kurtosis:                       7.251   Cond. No.                     7.24e+04
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.24e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

Any help would be appreciated!

$\endgroup$
10
  • $\begingroup$ This may be irrelevant, but I don't think you need to include the 1 with the formula API. $\endgroup$ – Demetri Pananos Oct 28 '16 at 18:51
  • 2
    $\begingroup$ You can have multicollinearity between a single covariate and the intercept, but no one ever cares. I can't tell if that's what you have or not. I also note that the relationship appears to be curvilinear & probably heteroscedastic. $\endgroup$ – gung - Reinstate Monica Oct 28 '16 at 18:59
  • 1
    $\begingroup$ Do I understand correctly that you have a regression w/ only 1 x-variable, which is centered, & it tells you you have a large condition number? That is pretty strange. There is probably something wrong w/ Python here. $\endgroup$ – gung - Reinstate Monica Oct 28 '16 at 20:44
  • 3
    $\begingroup$ My guess is that this is not related to collinearity but to the scaling of the explanatory variables, i.e. an eigenvalue is large because the variable is large, and some variables, like constant, are small. (statsmodels reports the condition number of the design matrix and not of a standardized design matrix.) $\endgroup$ – Josef Oct 28 '16 at 20:54
  • 1
    $\begingroup$ @user333700 Thanks for the info. I scaled each of my explanatory variables down by a factor of 1000, and that removed the condition number warning. This issue might be related as well: github.com/statsmodels/statsmodels/issues/553 $\endgroup$ – pst0102 Oct 28 '16 at 23:56
0
$\begingroup$

I found this page in a search, because I had the same question, but I think I have figured out what's going on.

First, a demonstration of the problem:

import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
x = np.arange(1000.,1030.,1.)   
y = 0.5*x
X = sm.add_constant(x)
plt.plot(x, y,'x')
plt.show()

enter image description here

mod_ols = sm.OLS(y, X)
res_ols = mod_ols.fit()
print(res_ols.summary())

enter image description here

Notice the very high condition number of 1.19e+05. This is because we're fitting a line to the points and then projecting the line all the way back to the origin (x=0) to find the y-intercept. That y-intercept will be very sensitive to small movements in the data points. The condition number takes into account high sensitivity in either fitted parameter to the input data, hence the high condition number when all of the data are far to one side of x=0.

To solve this, we simply center the x-values:

x -= np.average(x)
X = sm.add_constant(x)
plt.plot(x, y,'x')
plt.show()

enter image description here

enter image description here

The condition number is now greatly reduced to only 8.66. Notice that the fitted slope and calculated R**2 etc. are unchanged.

My conclusion: in the case of regression against a single variable, don't worry about the condition number UNLESS you care about the sensitivity of your y-intercept to the input data. If you do, then center the x-values first.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.