Assessing multicollinearity of factors

Question

I have a multifactor model (with 7 factors currently) and 754018 observations. In order to check for multicollinearity issues as the model grows I wrote an R script to compute a correlation matrix from the factors (so with 7 factors it's a 7x7 matrix). I then apply the function:

$$\frac{r}{(1-r^2) / (N-2)}$$

where $N = 754018$ and $r$ is the sample correlation in order to get a test statistic according to http://faculty.vassar.edu/lowry/ch4apx.html.

Then I get a corresponding p-value, and display those factor pairs (and their sample correlation) whose correlation p-value is less than $0.05$.

After running this I get 12 pairs displayed! With 7 factors the total number of possible pairs is 21 so this is pretty bad. Out of these 12, however, only 4 of them have correlations above .1 and the rest have sample correlations of around .02 or .01...from a practical standpoint should I worry about any nonzero correlation with very small pvalue (as in all 12 pairs) or only those with small pvalue AND high sample correlation? As in maybe only those 4? If the latter, are there empirical ways of choosing a threshold sample correlation?

Thanks

Answer 1

Suppose that we have a matrix X of n observations on k factors.

Suppose that W is the k by k matrix of the eigenvectors of the k by k correlation matrix V of X.

Since V is symmetric and positive definite, all its eigenvalues are real and positive.

Then the n by k matrix P of the principal components of V is such that: $\text{P=XW}$.

So if we regress on P instead of regressing on X, we obtain a vector of k coefficients $\gamma$, say, such that

$$P\gamma=\left(XW\right)\gamma=X\left(W\gamma\right)$$

Since P is orthogonal, the regression on P is well behaved, unless the rank of V is less than k.

But since the factors are multicollinear, $\beta=W\gamma$ is not well defined.

In order to approximate X, we can use the matrix $W_{\left(j\right)}$ of the first j eigenvectors of V, which define the first j columns of P, renamed $P_{\left(j\right)}$.

Since W is orthonormal, we have that $W^{-1}=W^{'}$, so that $X=PW^{'}$. The corresponding approximation for X would be

$$X_{\left(j\right)}=P_{\left(j\right)}W_{\left(j\right)}^{'}$$

This approximation could be such that the k columns of matrix $X_{\left(j\right)}$ are no longer multicollinear.

Accordingly, if we name $\gamma_{\left(j\right)}$ the first j coefficients of the vector $\gamma$, we have that

$$P_{\left(j\right)}\gamma_{\left(j\right)}=\left(X_{\left(j\right)}W_{\left(j\right)}\right)\gamma_{\left(j\right)}=X_{\left(j\right)}\left(W_{\left(j\right)}\gamma_{\left(j\right)}\right)$$

Thus, if we define $\beta_{\left(j\right)}=W_{\left(j\right)}\gamma_{\left(j\right)}$, we have a vector of k well behaved coefficients, in principle.

The proportion of the variance of X explained by the first j eigenvectors of V is given by the proportion of the cumulative sum of the corresponding eigenvalues of V to the sum of all eigenvalues of V. Since a few of the largest eigenvalues of V form the largest proportion of the sum of all eigenvalues of V when multicollinearity is a problem, we must choose the j eigenvectors corresponding to the largest j eigenvalues of V to approximate X.