Assessing multicollinearity of factors

Question

I have a multifactor model (with 7 factors currently) and 754018 observations. In order to check for multicollinearity issues as the model grows I wrote an R script to compute a correlation matrix from the factors (so with 7 factors it's a 7x7 matrix). I then apply the function:

$$\frac{r}{(1-r^2) / (N-2)}$$

where $N = 754018$ and $r$ is the sample correlation in order to get a test statistic according to http://faculty.vassar.edu/lowry/ch4apx.html.

Then I get a corresponding p-value, and display those factor pairs (and their sample correlation) whose correlation p-value is less than $0.05$.

After running this I get 12 pairs displayed! With 7 factors the total number of possible pairs is 21 so this is pretty bad. Out of these 12, however, only 4 of them have correlations above .1 and the rest have sample correlations of around .02 or .01...from a practical standpoint should I worry about any nonzero correlation with very small pvalue (as in all 12 pairs) or only those with small pvalue AND high sample correlation? As in maybe only those 4? If the latter, are there empirical ways of choosing a threshold sample correlation?

Thanks

Answer 1

There are multiple issues here. Among them are:

Most central, I think, is the relation between correlation and collinearity. You can have very high collinearity without any high correlations. For an intuitive, extreme case, imagine that there are 10 independent variables; nine of them are perfectly orthogonal (within sampling error) and the tenth is the sum of the other nine. You will then have no very high correlation, but perfect collinearity. You can also have sets of IVs that are related to each other.

The key point is that collinearity is a relationship among sets of IVs, not necessarily pairs.

A second point relates to hypothesis testing and p values. People who have read my posts know that I am a fairly extreme anti-p-value person, but here, the whole hypothesis testing apparatus is not appropriate. You have no hypothesis to test. The test that you did tests whether the correlations are significantly different from 0. But .... who cares? First, you have ~750,000 observations. With that data set size, even a tiny correlation will be significantly different from 0, but that won't have any bad effects on a regression. Second, even with a smaller data set, the key question is not whether the correlations are different from 0, but how big the condition indexes are (I prefer these to VIFs, but you can use VIFs if you like).

(As an aside, in Belsley's books he shows that you can have fairly large correlations and not have problematic collinearity. I don't have access to his books any more, but I read them for my dissertation).

Third, you say you started with seven factors. You don't describe them. Maybe these are continuous independent variables. But, from my experience, the word "factor" in a context like this either means a) Results of a factor analysis or b) Categorical variables. If it's a) then, for the most common types of FA, the factors are orthogonal. They can't be collinear. If b) then I am not sure what you did.

The comments (above) raise some other issues, too, but I think that's enough.

Answer 2

Suppose that we have a matrix X of n observations on k factors.

Suppose that W is the k by k matrix of the eigenvectors of the k by k correlation matrix V of X.

Since V is symmetric and positive definite, all its eigenvalues are real and positive.

Then the n by k matrix P of the principal components of V is such that: $\text{P=XW}$.

So if we regress on P instead of regressing on X, we obtain a vector of k coefficients $\gamma$, say, such that

$$P\gamma=\left(XW\right)\gamma=X\left(W\gamma\right)$$

Since P is orthogonal, the regression on P is well behaved, unless the rank of V is less than k.

But since the factors are multicollinear, $\beta=W\gamma$ is not well defined.

In order to approximate X, we can use the matrix $W_{\left(j\right)}$ of the first j eigenvectors of V, which define the first j columns of P, renamed $P_{\left(j\right)}$.

Since W is orthonormal, we have that $W^{-1}=W^{'}$, so that $X=PW^{'}$. The corresponding approximation for X would be

$$X_{\left(j\right)}=P_{\left(j\right)}W_{\left(j\right)}^{'}$$

This approximation could be such that the k columns of matrix $X_{\left(j\right)}$ are no longer multicollinear.

Accordingly, if we name $\gamma_{\left(j\right)}$ the first j coefficients of the vector $\gamma$, we have that

$$P_{\left(j\right)}\gamma_{\left(j\right)}=\left(X_{\left(j\right)}W_{\left(j\right)}\right)\gamma_{\left(j\right)}=X_{\left(j\right)}\left(W_{\left(j\right)}\gamma_{\left(j\right)}\right)$$

Thus, if we define $\beta_{\left(j\right)}=W_{\left(j\right)}\gamma_{\left(j\right)}$, we have a vector of k well behaved coefficients, in principle.

The proportion of the variance of X explained by the first j eigenvectors of V is given by the proportion of the cumulative sum of the corresponding eigenvalues of V to the sum of all eigenvalues of V. Since a few of the largest eigenvalues of V form the largest proportion of the sum of all eigenvalues of V when multicollinearity is a problem, we must choose the j eigenvectors corresponding to the largest j eigenvalues of V to approximate X.