Given a large amount of data, how would I determine the degree of collinearity between all the variables? Preferably without relying on calculating linear regression between a variable and every other variable? Specifically, I need to find the collinearity between any single variable and every single other variable.

For this instance, we'll say the data has 10k variables and 100 million lines. I don't need a software package per se, a link to a relevant paper is fine.

  • $\begingroup$ How much data have you got? The best method, in my opinion, is condition indexes and proportion of variance explained. $\endgroup$ – Peter Flom - Reinstate Monica Apr 19 '13 at 22:50
  • 1
    $\begingroup$ As Peter wrote, how big really? You can compute the condition number of your design matrix on a desktop for up to 20k variables in seconds. $\endgroup$ – user603 Apr 19 '13 at 23:15
  • $\begingroup$ @PeterFlom For the purpose of this question, I'll say the data has around 10k variables, and around 10 million points. I'll update my question. I'll also check out condition indexes. $\endgroup$ – saccharine Apr 20 '13 at 8:20
  • 1
    $\begingroup$ @user603 Around 10k variables and 100 million points. I somehow doubt that a 20k variable dataset can have the condition number computed in seconds, considering the harddrive can only read on the order of 100 mbs a second. $\endgroup$ – saccharine Apr 20 '13 at 8:21

The condition number is the statistic you want to look at. It's an exhaustive measure of colinearity of your design space and much cheaper to compute than the VIF. The formula is:

$$2\log\left(\frac{d_1(X_s)}{d_p(X_s)}\right) (1)$$

where $d_p$ ($d_1$) is the smallest (largest) singular value of $X_s$ and $X_s$ are the centred, re-scaled variables.

Reading your comment: this is what I was suspecting. To compute the condition # you don't need all 100e6 data points. Just samples, randomly, in the order of 100e3 (by experimenting with simulated datasets you can convince yourself that to get reliable results you should target about 5*k, where k is the # of non collinear variables so even 100e3 is very large already).

That should already give you a pretty good idea what variables are causing collinearity.

Also, you have specialized algorithm to compute the first and last few singular values and the last few singular vector only. There are algorithms do get these w/o computing the full SVD of $X$ (SVD-S). However, I don't know if these are implemented in R so to make the example below accessible I'll just use a small example and the classical SVD from R.

A high condition number (typically when the ratio (1) is larger than 10) tells you that $X_s'X_s$ is ill-conditioned, that there are components that can be re-written as (near) linear combination of the other variables. Below I give a brief (small scale) example of how SVD can be used to uncover such relations.

#non-ill conditioned part of the dataset.
#introduce a variable that causes x to be 
#ill conditioned.

#x is ill-conditioned: this ratio is larger 
#than 10. (step 1)

#check what is causing it: (step 2)
#you can write the last variable as (.23*x_1+.5*x_2-.45*x_3)/(-.7) [1]
#here the relation is exact because:
#is 0. if min(A$d)>0 then this gives you how much there is noise 
#there is arround [1].

#so I remove the last variable. (step 3)
#no more ill-condition.

This is for the linear algebra of the problem when there is a single variable causing the ill-regression. In most case you will have more than one (near) exact relationship and you will have to repeat steps 1 through 3. In practice, the computational specifics will depend on how clever is the approach you use to solve the SVD problem.

You can make yourself an idea of how many exact relationships there are in your dataset by computing


for all $j$'s. To do this you only need the singular values which you can get at cost $O(p^2)$

  • $\begingroup$ Can you link me something to read more on condition # and how it would help determining collinearity? Wikipedia seems to indicate that it doesn't provide the answers I need, which is the degree of collinearity between each variable. I'm attempting to simply group highly collinear variables together. $\endgroup$ – saccharine Apr 20 '13 at 8:46
  • 1
    $\begingroup$ Read Belsley: Conditioning diagnostics, collinearity and weak data in regression. Or, look up my dissertation Flom, Collinearity diagnostics for multiple regression: A Monte Carlo Study. Belsley also wrote another book that covers this, but is more general. $\endgroup$ – Peter Flom - Reinstate Monica Apr 20 '13 at 13:03
  • $\begingroup$ @user603 Can you explain what's going on in step 2? How exactly do you know that the last variable is causing the issue? Specifically, how did you arrive at "round(A$v[,ncol(A$v)],2)"? You're taking the right singular SVD of the last variable? How did you know to look for that? $\endgroup$ – saccharine Apr 22 '13 at 18:24
  • $\begingroup$ @PeterFlom Do you have a version of your paper open for downloading anywhere? Fordham Univ won't let me access it. $\endgroup$ – saccharine Apr 22 '13 at 18:30
  • $\begingroup$ @saccharine: by definition, the $j^{th}$ singular value is the mean of the squared deviations of the data projected on the $j^{th}$ right singular vector. In R, the singular vectors are ordered by decreasing values of the associated singular values. So, the $j^{th}$ singular vector gives you the linear combination of your data yielding the $j^{th}$ largest deviations. Is this clear? $\endgroup$ – user603 Apr 22 '13 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.