# How to identify which variables are collinear in a singular regression matrix? [duplicate]

I have a matrix on which I am performing a zero inflated regression model but it return an error indicating collinearity between some of the variables

zinf1 = zeroinfl(count ~ origin + variable + gene, data = count_FGT_free, dist = "negbin")
Warning message:
In value[[3L]](cond) :
Lapack routine dgesv: system is exactly singular: U[590,590] = 0FALSE


What is the best way to identify which variables are the cause of this?

You can use the QR decomposition with column pivoting (see e.g. "The Behavior of the QR-Factorization Algorithm with Column Pivoting" by Engler (1997)). As described in that paper, the pivots give an ordering of the columns by "most linearly independent". Assuming we've computed the rank of the matrix already (which is a fair assumption since in general we'd need to do this to know it's low rank in the first place) we can then take the first $$\text{rank}(X)$$ pivots and should get a full rank matrix.

Here's an example.

set.seed(1)
n <- 50
inputs <- matrix(rnorm(n*3), n, 3)
x <- cbind(
inputs[,1], inputs[,2], inputs[,1] + inputs[,2],
inputs[,3], -.25 * inputs[,3]
)
print(Matrix::rankMatrix(x))  # 5 columns but rank 3

cor(x)  # only detects the columns 4,5 collinearity, not 1,2,3
svd(x)$d # two singular values are numerically zero as expected qr.x <- qr(x) print(qr.x$$pivot) rank.x <- Matrix::rankMatrix(x) print(Matrix::rankMatrix(x[,qr.x$$pivot[1:rank.x]])) # full rank  Another comment on issues with just using pairwise correlation is that two columns having a perfect correlation doesn't even guarantee the matrix is low rank. As an example: set.seed(1) x <- rnorm(n) x <- cbind(x, x + 1) print(Matrix::rankMatrix(x)) cor(x)  These two columns are perfectly correlated, but since the constant vector is not in their span, it doesn't actually affect the rank. If there also was an intercept column then this matrix would indeed be rank $$2$$ (almost surely). • Thank you for this. Should this work one count data with categorical variables? I ask as qr(my matrix) returns errors NA/NaN/Inf in foreign function call (arg 1) – RMM Jul 9 at 7:34 • @RMM this works on a numeric matrix and doesn't care if it happens to represent encodings of categorical variables. Is there a chance you're doing this on your dataframe with factors still in it rather than a purely numeric dataframe or matrix? – jld Jul 9 at 15:03 • I am indeed, shall I just transform all the categories into numerical representations? – RMM Jul 9 at 15:14 • @RMM yeah the QR decomposition requires a numeric matrix – jld Jul 9 at 16:21 It looks like you have perfect collinearity among one of the pairs of your 3 independent variables. Run a correlation matrix, and check which pair has a correlation of exactly 1. In R, cor(count_FGT_free)  You could also create a smaller dataframe with just those three variables if count_FGT_free is large. • This won’t always catch it because you could have a low rank matrix without any pair of columns being exactly collinear. Eg make a matrix with two iid Gaussian columns and set the third to the sum of the first two. This matrix will be low rank because$(1,1,-1)^T\$ is a nontrivial null vector but no pair of columns has a perfect correlation – jld Jul 8 at 16:46
• Good point! Thanks. That being said, I would probably first do a cor(x) check to see if I could quickly catch the collinearity before diving into a QR decomposition. – Alex Jul 8 at 21:42
• Fair enough, that is quick starting point – jld Jul 9 at 4:19
• Can this be done on count data with categorical variables? – RMM Jul 9 at 7:35
• You'd have to turn the categorical variables into dummies, I believe. – Alex Jul 9 at 14:12