Return to Revisions

Detailed explanation of the problem :

In the case of X being near-singular (high colinearity/covariance between features), different issues where coming both from scipy.linalg.lstsq() and sklearn.linear_model.LinearRegession()

Source of error 1 : As @SextusEmpiricus explained, the matrix being near-singular leads to rounding errors that impact enormously the final predictions. In this sense, scipy.linalg.lstsq() is silently failing WITHOUT raising any warning or error.

Source of error 2 :The matrix coming from pandas was F-contiguous. Sklearn converts it to C-contiguous before calling scipy.linalg.lstsq() and then use the predict() by using a matrux multiplication right from the F-contiguous array. This lead to another layer of rounding errors. I opened another question here on StackOverflow

Source of error 3 : The first thing that LinearRegression() is doing is to center the dataframe. This goes badly in my case, I still struggle to understand why exactly.

Note : Please note that these rounding errors also depends on CPUs and hardware, which makes it even hard to achieve reproducibility.

(Partial) Work-Around :

To work around the sklearn problems, one can :

• Ensure input matrix/array are C-contiguous
• Stop rely on LinearRegression's fit_intercept=True but instead center data manually first :

for seed in range(1000) :
    np.random.seed(seed)
    s = pd.Series(np.random.normal(10, 1, size=1_000))

    l_com = np.arange(100)
    df_Xy = pd.concat([s.ewm(com=com).mean() for com in l_com], axis=1)
    df_Xy['y'] = s.shift(-1)
    df_Xy.dropna(inplace=True)

    X = np.ascontiguousarray(df_Xy[l_com].values)
    y = np.ascontiguousarray(df_Xy.y.values)

    X_offset = X.mean(axis=0)
    y_offset = y.mean()

    X_centered = X - X_offset
    y_centered = y - y_offset

    model = LinearRegression(fit_intercept=False) # We don't rely on sklearn fit_intercept anymore
    model.fit(X_centered, y_centered)
    assert model.score(X_centered, y_centered) > 0 # ALL GOOD

Moving forward / Long-term Solution :

1. I opened an issue in scipy Github to raise a Warning in scipy.linalg.lstsq when the X matrix is near-singular.

2. I opened an issue in sklearn project on Github, about inconsistency between C-cont vs F-cont arrays