I observed than when using Lasso regression and KFold crossvalidation with my data, predicted values show negative correlation with observed values.

I tried to replicate the problem with a randomly generated dataset roughly similar to the one I am working on (10 features, 5 of which are correlated with the response variable and 500 observations) and see something similar.

#Lasso example
from sklearn.linear_model import Lasso
from sklearn.model_selection import KFold
from scipy.stats import pearsonr
import numpy as np

#create random features
rng = np.random.RandomState(seed=42)
X = rng.randn(2500).reshape(500, 5)

#create additional correlated random features

#create covariance matrix
covs = np.full((6,6), 0.8) #set correlation (== covariance since means are 0 and sd is one) to 0.8 for all of these variables 
np.fill_diagonal(covs, 1)

means = np.zeros(6)

X_correlated = rng.multivariate_normal(means, covs, 500)

#append all of the correlated features but one to X
X = np.c_[X, X_correlated[:,:-1]] 

#set the last of the correlated features as Y
Y = X_correlated[:,-1]

#instantiate regressor and cv object
cv = KFold(100)
reg = Lasso(random_state=42)

#create arrays to save predicted (and observed) Y values
pred = np.array([])
obs = np.array([])

#run cross validation
for train, test in cv.split(X):

    #fit regressor
    reg.fit(X[train], Y[train])

    #append predicted and observed values to the arrays
    pred = np.r_[pred, reg.predict(X[test])]
    obs = np.r_[obs, Y[test]]

#test correlation
pearsonr(pred, obs)

#r = -0.424, p < 10**(-22)

In my hands, using L2 penalized regression (Ridge()) or LinearRegression() in the same setting produces results that make sense. Does anyone have any idea what I am doing wrong?


migrated from stackoverflow.com Apr 1 at 17:31

This question came from our site for professional and enthusiast programmers.

  • 1
    $\begingroup$ +1. I suspect you just discovered that LASSO estimates are not unbiased estimates but actually exhibit shrinkage towards zero. For $N=500K$ we get correlation $-0.013$, for $N=50K$ we get correlation $-0.047$, for $N=5K$ we get correlation $-0.143$ and for $N=500$ (the case you show) we get correlation $-0.446$. i.e. we have a clear sample-size influence at this! $\endgroup$ – usεr11852 Apr 1 at 22:30
  • $\begingroup$ Thanks! Interesting observation that the correlation is dependent on the sample size. Still I cannot quite make sense of it, because if all coefficients get shrunk to zero, the output would equal the intercept and be constant. Furthermore, I noticed that when leaving Leave one out cross validation, I get correlation -1. $\endgroup$ – M.Zatzenbreck Apr 2 at 11:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.