# Assessing multicollinearity of factors

I have a multifactor model (with 7 factors currently) and 754018 observations. In order to check for multicollinearity issues as the model grows I wrote an R script to compute a correlation matrix from the factors (so with 7 factors it's a 7x7 matrix). I then apply the function:

$$\frac{r}{(1-r^2) / (N-2)}$$

where $N = 754018$ and $r$ is the sample correlation in order to get a test statistic according to http://faculty.vassar.edu/lowry/ch4apx.html.

Then I get a corresponding p-value, and display those factor pairs (and their sample correlation) whose correlation p-value is less than $0.05$.

After running this I get 12 pairs displayed! With 7 factors the total number of possible pairs is 21 so this is pretty bad. Out of these 12, however, only 4 of them have correlations above .1 and the rest have sample correlations of around .02 or .01...from a practical standpoint should I worry about any nonzero correlation with very small pvalue (as in all 12 pairs) or only those with small pvalue AND high sample correlation? As in maybe only those 4? If the latter, are there empirical ways of choosing a threshold sample correlation?

Thanks

-
Correlations by themselves don't tell us a lot about multicollinearity; hypothesis tests about correlations tell us nothing at all. What is the condition number of the matrix? What about the VIFs in the regression? –  whuber Dec 23 '11 at 17:23
@whuber correlation between two independent variables isn't useful for multicollinearity? I thought that's what multicollinearity was all about...and that if two features were too correlated that this would inflate standard errors being it harder to differentiate their effect on the independent? Suppose I find VIFs..then do I have to do this for each factor (removing it one by one) to compare? Could I just look at the adjusted R-squared for every factor thrown in? –  Palace Chan Dec 23 '11 at 19:05
@Adam The multifactor model is for prediction. Doesn't multicollinearity affect the estimation of parameters? –  Palace Chan Dec 23 '11 at 19:08
Palace, your test merely establishes that some correlations are nonzero. Pairwise correlations will be problematic for regression only when their magnitudes are close to 1, which is not at all the same as being different from 0! Moreover, p-values are completely meaningless: you're not trying to make inferences about how the independent variables are correlated within a population; all you care about for regression is how the IVs in your particular sample happen to be correlated. The effects of their mutual correlations are better assessed with diagnostics like the VIF. –  whuber Dec 23 '11 at 19:45

Suppose that we have a matrix X of n observations on k factors.

Suppose that W is the k by k matrix of the eigenvectors of the k by k correlation matrix V of X.

Since V is symmetric and positive definite, all its eigenvalues are real and positive.

Then the n by k matrix P of the principal components of V is such that: $\text{P=XW}$.

So if we regress on P instead of regressing on X, we obtain a vector of k coefficients $\gamma$, say, such that

$$P\gamma=\left(XW\right)\gamma=X\left(W\gamma\right)$$

Since P is orthogonal, the regression on P is well behaved, unless the rank of V is less than k.

But since the factors are multicollinear, $\beta=W\gamma$ is not well defined.

In order to approximate X, we can use the matrix $W_{\left(j\right)}$ of the first j eigenvectors of V, which define the first j columns of P, renamed $P_{\left(j\right)}$.

Since W is orthonormal, we have that $W^{-1}=W^{'}$, so that $X=PW^{'}$. The corresponding approximation for X would be

$$X_{\left(j\right)}=P_{\left(j\right)}W_{\left(j\right)}^{'}$$

This approximation could be such that the k columns of matrix $X_{\left(j\right)}$ are no longer multicollinear.

Accordingly, if we name $\gamma_{\left(j\right)}$ the first j coefficients of the vector $\gamma$, we have that

$$P_{\left(j\right)}\gamma_{\left(j\right)}=\left(X_{\left(j\right)}W_{\left(j\right)}\right)\gamma_{\left(j\right)}=X_{\left(j\right)}\left(W_{\left(j\right)}\gamma_{\left(j\right)}\right)$$

Thus, if we define $\beta_{\left(j\right)}=W_{\left(j\right)}\gamma_{\left(j\right)}$, we have a vector of k well behaved coefficients, in principle.

The proportion of the variance of X explained by the first j eigenvectors of V is given by the proportion of the cumulative sum of the corresponding eigenvalues of V to the sum of all eigenvalues of V. Since a few of the largest eigenvalues of V form the largest proportion of the sum of all eigenvalues of V when multicollinearity is a problem, we must choose the j eigenvectors corresponding to the largest j eigenvalues of V to approximate X.

-
(The $\TeX$ will work much better when delimited with dollar signs (\$).) You describe "PCA regression," but I have some doubts about its applicability, in particular concerning the last paragraph. What matters for regression are the partial correlations between the independent variables and the dependent variable. It is possible--even likely--that some of the principal components with small eigenvalues may be much better correlated with the dependent variable than some of the PCs with larger EVs. Thus, no criterion based solely on the sizes of the EVs can be generally appropriate. –  whuber Dec 24 '11 at 20:00