I have defined two numpy arrays X and y as

(array([[ 1.,  1.,  1.,  4.],
        [ 2.,  2.,  2.,  1.],
        [ 3.,  3.,  3.,  7.],
        [-1., -1., -1.,  2.],
        [-2., -2., -2., 13.],
        [-3., -3., -3.,  7.]])
array([[ 7], [ 7], [16], [-1], [ 7], [-2]])

I am trying to solve for y = Xw. It's clear that rank(X) = 2 since we have 2 independent variables. But how is sklearn.LinearRegression able to fit this dataset?

# Applying linear regression model (Normal equation)
reg = LinearRegression().fit(X, y)    
print('rank', reg.rank_)
print('score', reg.score(X, y))
print('w', reg.coef_,)
print('bias', reg.intercept_)

It gives the perfect solution. If it solves the equation using normal method, shouldn't it throw the exception that $X^T X$ is non invertible?

rank 2
score 1.0
w [[1. 1. 1. 1.]]
bias [-2.66453526e-15]

This is the expected behavior.

I'll use the same symbols from the scipy documentation.

From the sklearn documentation, we read that LinearRegression is just a wrapper for scipy.linalg.lstsq. Reading the documentaiton for scipy.linalg.lstsq, we find that this function carries out a specific minimization:

Compute a vector x such that the 2-norm |b - A x| is minimized.

In other words, even if there are many $x$ that solve the linear system $Ax=b$, the scipy.linalg.lstsq function returns an $x$ that minimizes $\| b - Ax\|_2$.

In your case, the matrix $A$ is rank-deficient, and there are many $x$ that minimize $\| b - Ax \|_2$. In this case, LAPACK is used to compute the solution which minimizes both $\| x \|_2$ and $\| b - Ax \|_2$.

These LAPACK functions have an overview here: http://netlib.org/lapack/lug/node27.html

  • $\begingroup$ How does lstsq do it then? How does it handle non-invertibility? $\endgroup$ – Shashwat Jun 20 '20 at 18:10
  • 2
    $\begingroup$ You don't have to invert a matrix to solve a linear system; it just happens that if there is a unique solution to a linear system, then matrix inversion is an obvious approach. Specifically, scipy.linalg.lstsq is calling LAPACK routines which have an overview here: netlib.org/lapack/lug/node27.html $\endgroup$ – Sycorax Jun 20 '20 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.