1
$\begingroup$

In https://www.statsmodels.org/0.8.0/examples/notebooks/generated/ols.html, if you scroll down to the section "Multicollinearity" (bolded), it shows that the condition number of $X$ is $4.86e+09$. Then right below that, it says

Condition number One way to assess multicollinearity is to compute the condition number. Values over 20 are worrisome (see Greene 4.9). The first step is to normalize the independent variables to have unit length

and then they compute another condition number using

norm_x = X.values
for i, name in enumerate(X):
    if name == "const":
        continue
    norm_x[:,i] = X[name]/np.linalg.norm(X[name])
norm_xtx = np.dot(norm_x.T,norm_x)

eigs = np.linalg.eigvals(norm_xtx)
condition_number = np.sqrt(eigs.max() / eigs.min())
print(condition_number)

and the result comes out to be 56240.8714071 .

I'm confused about several things here.

(1) Are we more interested in the condition number of $X$ or $X^TX$ when assessing the conditioning of the system for OLS? The linear system of equations is $X \beta = y$, but the normal equations are $X^TX\beta = X^Ty$, which is also a linear system of equations. I'm assuming this depends on the method since some methods for solving OLS don't even form the normal equations. I think statsmodels in Python uses SVD, which just performs the decomposition for $X$, so what's the point of computing the condition number for a normalized version of $X^TX$ here?

(2) Why are they normalizing the independent variables before computing the condition number?

$\endgroup$
7
  • 1
    $\begingroup$ Does this answer your question? Relationship between SVD and PCA. How to use SVD to perform PCA? Your question is essentially the close relationship between the covariance matrix $X^TX$ used in PCA and the SVD discussed on that page. Note the part of the answer about the importance of centering in this context, the first step in normalization. $\endgroup$
    – EdM
    May 7, 2021 at 14:39
  • $\begingroup$ For better advice (than the stuff you quote) on detecting and diagnosing collinearity, see our thread at stats.stackexchange.com/questions/173665. $\endgroup$
    – whuber
    May 7, 2021 at 15:00
  • $\begingroup$ @EdM I don't think so because I'm referring to condition number specifically, and the condition number for $X$ is not equivalent to $X^TX$. $\endgroup$
    – 24n8
    May 7, 2021 at 16:19
  • $\begingroup$ @whuber Yes, I think normally I would use VIF, but I think condition number as some use when it comes to just a quick check for multicollinearity, and perhaps use that quick check to decide if we should use OLS or something like Ridge. With VIF we have to separately regress $p$ variables onto the other $p - 1$ variables, so it's not as of a quick check $\endgroup$
    – 24n8
    May 7, 2021 at 16:23
  • 3
    $\begingroup$ The condition number of $X^\prime X$ is the square of the condition number of $X,$ so they are equivalent: they're just different ways of expressing the same property of $X.$ In the link I gave, the formula for VIF does not require regressing all variables on all others: it needs only the diagonal elements of the the inverse of $X^\prime X.$ $\endgroup$
    – whuber
    May 7, 2021 at 16:35

1 Answer 1

2
$\begingroup$

The condition number for the design matrix X as reported by statsmodels is an indicator for numerical problems that can be caused by either multicollinearity or bad scaling.

The literature on multicollinearity usually uses standardized variables that remove scaling problems and focus on multicollinearity. The old low recommended thresholds are for the standardized version.

Statsmodels uses the non-standardized version of the design matrix to signal that there might be problems also in the case when the design matrix is badly scaled even without multicollinearity. There are several questions on crossvalidated and stack overflow related to scaling problems, for example Regression model constant causes multicollinearity warning, but not in standardized model .

There was also a debate about whether the constant should be included in the collinearity measures. Some authors use the correlation matrix, i.e. demeaned and standardized, while Belseley, Kuh and Welch argue for including constant to signal problems with the level of the variables.

The computation of the variance inflation factor of the current function in statsmodels also uses the user provided design matrix. This means that in the usual case the vif for the constant is computed.
Two cases of possible problems with the constant are variables with very little fluctuations that are almost constant, i.e. have small variance relative to (nonzero) mean, and secondly cases like dummy variable trap where users do not drop a reference category and a categorical variable is "collinear" with the constant. The correlation matrix of the slope variables would not detect those problems.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.