# sklearn LinearRegression handles rank deficient matrix

I have defined two numpy arrays X and y as

X
(array([[ 1.,  1.,  1.,  4.],
[ 2.,  2.,  2.,  1.],
[ 3.,  3.,  3.,  7.],
[-1., -1., -1.,  2.],
[-2., -2., -2., 13.],
[-3., -3., -3.,  7.]])
y
array([[ 7], [ 7], [16], [-1], [ 7], [-2]])


I am trying to solve for y = Xw. It's clear that rank(X) = 2 since we have 2 independent variables. But how is sklearn.LinearRegression able to fit this dataset?

# Applying linear regression model (Normal equation)
reg = LinearRegression().fit(X, y)
print('rank', reg.rank_)
print('score', reg.score(X, y))
print('w', reg.coef_,)
print('bias', reg.intercept_)


It gives the perfect solution. If it solves the equation using normal method, shouldn't it throw the exception that $$X^T X$$ is non invertible?

rank 2
score 1.0
w [[1. 1. 1. 1.]]
bias [-2.66453526e-15]


This is the expected behavior.

I'll use the same symbols from the scipy documentation.

From the sklearn documentation, we read that LinearRegression is just a wrapper for scipy.linalg.lstsq. Reading the documentaiton for scipy.linalg.lstsq, we find that this function carries out a specific minimization:

Compute a vector x such that the 2-norm |b - A x| is minimized.

In other words, even if there are many $$x$$ that solve the linear system $$Ax=b$$, the scipy.linalg.lstsq function returns an $$x$$ that minimizes $$\| b - Ax\|_2$$.

In your case, the matrix $$A$$ is rank-deficient, and there are many $$x$$ that minimize $$\| b - Ax \|_2$$. In this case, LAPACK is used to compute the solution which minimizes both $$\| x \|_2$$ and $$\| b - Ax \|_2$$.

These LAPACK functions have an overview here: http://netlib.org/lapack/lug/node27.html

• How does lstsq do it then? How does it handle non-invertibility? Jun 20, 2020 at 18:10
• You don't have to invert a matrix to solve a linear system; it just happens that if there is a unique solution to a linear system, then matrix inversion is an obvious approach. Specifically, scipy.linalg.lstsq is calling LAPACK routines which have an overview here: netlib.org/lapack/lug/node27.html
– Sycorax
Jun 20, 2020 at 18:34