Return to Revisions

Detailed explanation of the problem :

In the case of X being near-singular, different issues where coming both from scipy.linalg.lstsq() and sklearn.linear_model.LinearRegession()

Source of error 1 : The matrix coming from pandas was F-contiguous (and not C-contiguous) which leads to bad rounding errors. I opened another question here on StackOverflow

Source of error 2 : The first thing that LinearRegression() is doing is to center the dataframe. This goes badly in my case, I still struggle to understand why exactly.

Source of error 3 : In case of near-singular matrix, scipy.linalg.lstsq() can silently fails WITHOUT raising any warning or error.

(Partial) Work-Around :

To work around the sklearn problems, one can :

• Ensure input matrix/array are C-contiguous
• Stop rely on LinearRegression's fit_intercept=True but instead center data manually first :

for seed in range(1000) :
    np.random.seed(seed)
    s = pd.Series(np.random.normal(10, 1, size=1_000))

    l_com = np.arange(100)
    df_Xy = pd.concat([s.ewm(com=com).mean() for com in l_com], axis=1)
    df_Xy['y'] = s.shift(-1)
    df_Xy.dropna(inplace=True)

    X = np.ascontiguousarray(df_Xy[l_com].values)
    y = np.ascontiguousarray(df_Xy.y.values)

    X_offset = X.mean(axis=0)
    y_offset = y.mean()

    X_centered = X - X_offset
    y_centered = y - y_offset

    model = LinearRegression(fit_intercept=False) # We don't rely on sklearn fit_intercept anymore
    model.fit(X_centered, y_centered)
    assert model.score(X_centered, y_centered) > 0 # ALL GOOD

Moving forward / Long-term Solution :

I will :

1. Open an issue in scipy Github to raise a Warning in scipy.linalg.lstsq when the X matrix is near-singular.

2. Open an issue in sklearn Github to always convert input matrix as C-contiguous + raise an issue when fit_intercept=True and data is near-singular (?) (unsure)