Cause of a high condition number in a python statsmodels regression?

Question

I'm pretty new to regression analysis, and I'm using python's statsmodels to look at the relationship between GDP/health/social services spending and health outcomes (DALYs) across the OECD. Just to give an idea of the data I'm using, this is a scatter matrix with the diagonal being the kernel density estimate:

When I run a simple regression of dalyrate on social_exp, the result shows a warning about a high condition number:

import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
import statsmodels.api as sm

poly_1 = smf.ols(formula='dalyrate ~ 1 + social_exp', data=model_df, missing='drop').fit()
print poly_1.summary()

                            OLS Regression Results                            
==============================================================================
Dep. Variable:               dalyrate   R-squared:                       0.253
Model:                            OLS   Adj. R-squared:                  0.248
Method:                 Least Squares   F-statistic:                     46.85
Date:                Fri, 28 Oct 2016   Prob (F-statistic):           2.30e-10
Time:                        12:56:43   Log-Likelihood:                -1336.8
No. Observations:                 140   AIC:                             2678.
Df Residuals:                     138   BIC:                             2683.
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept   2.705e+04    635.828     42.541      0.000      2.58e+04  2.83e+04
social_exp    -0.9303      0.136     -6.845      0.000        -1.199    -0.662
==============================================================================
Omnibus:                       34.504   Durbin-Watson:                   0.907
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               78.046
Skew:                           1.017   Prob(JB):                     1.13e-17
Kurtosis:                       6.039   Cond. No.                     1.03e+04
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 1.03e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

From what I've read, multicollinearity shouldn't be an issue with a single variable regression, so what could be causing this problem? It seems like both the gdp_cap and dalyrate variables are positively skewed in the scatter matrix, so do I need to normalize or standardize the data in order to prevent this warning? It would be ideal if I didn't need to because the units on the coefficient are helpful for interpreting the results, but obviously I'd do so if necessary.

I also get a similar warning with a multivariate regression:

poly_2 = smf.ols(formula='dalyrate ~ 1 + social_exp + health_exp + gdp_cap', data=model_df, missing='drop').fit()
print poly_2.summary()


                            OLS Regression Results                            
==============================================================================
Dep. Variable:               dalyrate   R-squared:                       0.406
Model:                            OLS   Adj. R-squared:                  0.393
Method:                 Least Squares   F-statistic:                     30.98
Date:                Fri, 28 Oct 2016   Prob (F-statistic):           2.51e-15
Time:                        13:04:35   Log-Likelihood:                -1320.8
No. Observations:                 140   AIC:                             2650.
Df Residuals:                     136   BIC:                             2661.
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept    2.86e+04    631.270     45.308      0.000      2.74e+04  2.98e+04
social_exp     0.0379      0.208      0.182      0.855        -0.373     0.449
health_exp    -1.1555      0.889     -1.300      0.196        -2.914     0.603
gdp_cap       -0.1412      0.053     -2.662      0.009        -0.246    -0.036
==============================================================================
Omnibus:                       53.794   Durbin-Watson:                   0.663
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              157.258
Skew:                           1.490   Prob(JB):                     7.11e-35
Kurtosis:                       7.251   Cond. No.                     7.24e+04
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The condition number is large, 7.24e+04. This might indicate that there are
strong multicollinearity or other numerical problems.

Any help would be appreciated!

Answer 1

I found this page in a search, because I had the same question, but I think I have figured out what's going on.

First, a demonstration of the problem:

import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
x = np.arange(1000.,1030.,1.)   
y = 0.5*x
X = sm.add_constant(x)
plt.plot(x, y,'x')
plt.show()

mod_ols = sm.OLS(y, X)
res_ols = mod_ols.fit()
print(res_ols.summary())

Notice the very high condition number of 1.19e+05. This is because we're fitting a line to the points and then projecting the line all the way back to the origin (x=0) to find the y-intercept. That y-intercept will be very sensitive to small movements in the data points. The condition number takes into account high sensitivity in either fitted parameter to the input data, hence the high condition number when all of the data are far to one side of x=0.

To solve this, we simply center the x-values:

x -= np.average(x)
X = sm.add_constant(x)
plt.plot(x, y,'x')
plt.show()

The condition number is now greatly reduced to only 8.66. Notice that the fitted slope and calculated R**2 etc. are unchanged.

My conclusion: in the case of regression against a single variable, don't worry about the condition number UNLESS you care about the sensitivity of your y-intercept to the input data. If you do, then center the x-values first.