How to identify which variables are collinear in a singular regression matrix? [duplicate]

Question

I have a matrix on which I am performing a zero inflated regression model but it return an error indicating collinearity between some of the variables

zinf1 = zeroinfl(count ~ origin + variable + gene, data = count_FGT_free, dist = "negbin")
Warning message:
In value[[3L]](cond) :
  Lapack routine dgesv: system is exactly singular: U[590,590] = 0FALSE

What is the best way to identify which variables are the cause of this?

Answer 1

You can use the QR decomposition with column pivoting (see e.g. "The Behavior of the QR-Factorization Algorithm with Column Pivoting" by Engler (1997)). As described in that paper, the pivots give an ordering of the columns by "most linearly independent". Assuming we've computed the rank of the matrix already (which is a fair assumption since in general we'd need to do this to know it's low rank in the first place) we can then take the first $\text{rank}(X)$ pivots and should get a full rank matrix.

Here's an example.

set.seed(1)
n <- 50
inputs <- matrix(rnorm(n*3), n, 3)
x <- cbind(
  inputs[,1], inputs[,2], inputs[,1] + inputs[,2],
  inputs[,3], -.25 * inputs[,3]
)
print(Matrix::rankMatrix(x))  # 5 columns but rank 3

cor(x)  # only detects the columns 4,5 collinearity, not 1,2,3
svd(x)$d  # two singular values are numerically zero as expected

qr.x <- qr(x)
print(qr.x$pivot)
rank.x <- Matrix::rankMatrix(x)
print(Matrix::rankMatrix(x[,qr.x$pivot[1:rank.x]]))  # full rank

Another comment on issues with just using pairwise correlation is that two columns having a perfect correlation doesn't even guarantee the matrix is low rank. As an example:

set.seed(1)
x <- rnorm(n)
x <- cbind(x, x + 1)
print(Matrix::rankMatrix(x))
cor(x)

These two columns are perfectly correlated, but since the constant vector is not in their span, it doesn't actually affect the rank. If there also was an intercept column then this matrix would indeed be rank $2$ (almost surely).

Answer 2

It looks like you have perfect collinearity among one of the pairs of your 3 independent variables. Run a correlation matrix, and check which pair has a correlation of exactly 1. In R,

cor(count_FGT_free)

You could also create a smaller dataframe with just those three variables if count_FGT_free is large.