1
$\begingroup$

I have defined two numpy arrays X and y as

X
(array([[ 1.,  1.,  1.,  4.],
        [ 2.,  2.,  2.,  1.],
        [ 3.,  3.,  3.,  7.],
        [-1., -1., -1.,  2.],
        [-2., -2., -2., 13.],
        [-3., -3., -3.,  7.]])
y
array([[ 7], [ 7], [16], [-1], [ 7], [-2]])

I am trying to solve for y = Xw. It's clear that rank(X) = 2 since we have 2 independent variables. But how is sklearn.LinearRegression able to fit this dataset?

# Applying linear regression model (Normal equation)
reg = LinearRegression().fit(X, y)    
print('rank', reg.rank_)
print('score', reg.score(X, y))
print('w', reg.coef_,)
print('bias', reg.intercept_)

It gives the perfect solution. If it solves the equation using normal method, shouldn't it throw the exception that $X^T X$ is non invertible?

rank 2
score 1.0
w [[1. 1. 1. 1.]]
bias [-2.66453526e-15]
$\endgroup$

1 Answer 1

6
$\begingroup$

This is the expected behavior.

I'll use the same symbols from the scipy documentation.

From the sklearn documentation, we read that LinearRegression is just a wrapper for scipy.linalg.lstsq. Reading the documentaiton for scipy.linalg.lstsq, we find that this function carries out a specific minimization:

Compute a vector x such that the 2-norm |b - A x| is minimized.

In other words, even if there are many $x$ that solve the linear system $Ax=b$, the scipy.linalg.lstsq function returns an $x$ that minimizes $\| b - Ax\|_2$.

In your case, the matrix $A$ is rank-deficient, and there are many $x$ that minimize $\| b - Ax \|_2$. In this case, LAPACK is used to compute the solution which minimizes both $\| x \|_2$ and $\| b - Ax \|_2$.

These LAPACK functions have an overview here: http://netlib.org/lapack/lug/node27.html

$\endgroup$
2
  • $\begingroup$ How does lstsq do it then? How does it handle non-invertibility? $\endgroup$
    – Shashwat
    Jun 20, 2020 at 18:10
  • 2
    $\begingroup$ You don't have to invert a matrix to solve a linear system; it just happens that if there is a unique solution to a linear system, then matrix inversion is an obvious approach. Specifically, scipy.linalg.lstsq is calling LAPACK routines which have an overview here: netlib.org/lapack/lug/node27.html $\endgroup$
    – Sycorax
    Jun 20, 2020 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.