# Predicted Ys for Lasso regression show negative correlation with observed Ys

I observed than when using Lasso regression and KFold crossvalidation with my data, predicted values show negative correlation with observed values.

I tried to replicate the problem with a randomly generated dataset roughly similar to the one I am working on (10 features, 5 of which are correlated with the response variable and 500 observations) and see something similar.

#Lasso example
from sklearn.linear_model import Lasso
from sklearn.model_selection import KFold
from scipy.stats import pearsonr
import numpy as np

#create random features
rng = np.random.RandomState(seed=42)
X = rng.randn(2500).reshape(500, 5)

#create additional correlated random features

#create covariance matrix
covs = np.full((6,6), 0.8) #set correlation (== covariance since means are 0 and sd is one) to 0.8 for all of these variables
np.fill_diagonal(covs, 1)

means = np.zeros(6)

X_correlated = rng.multivariate_normal(means, covs, 500)

#append all of the correlated features but one to X
X = np.c_[X, X_correlated[:,:-1]]

#set the last of the correlated features as Y
Y = X_correlated[:,-1]

#instantiate regressor and cv object
cv = KFold(100)
reg = Lasso(random_state=42)

#create arrays to save predicted (and observed) Y values
pred = np.array([])
obs = np.array([])

#run cross validation
for train, test in cv.split(X):

#fit regressor
reg.fit(X[train], Y[train])

#append predicted and observed values to the arrays
pred = np.r_[pred, reg.predict(X[test])]
obs = np.r_[obs, Y[test]]

#test correlation
pearsonr(pred, obs)

#r = -0.424, p < 10**(-22)



In my hands, using L2 penalized regression (Ridge()) or LinearRegression() in the same setting produces results that make sense. Does anyone have any idea what I am doing wrong?

## migrated from stackoverflow.comApr 1 at 17:31

This question came from our site for professional and enthusiast programmers.

• +1. I suspect you just discovered that LASSO estimates are not unbiased estimates but actually exhibit shrinkage towards zero. For $N=500K$ we get correlation $-0.013$, for $N=50K$ we get correlation $-0.047$, for $N=5K$ we get correlation $-0.143$ and for $N=500$ (the case you show) we get correlation $-0.446$. i.e. we have a clear sample-size influence at this! – usεr11852 Apr 1 at 22:30
• Thanks! Interesting observation that the correlation is dependent on the sample size. Still I cannot quite make sense of it, because if all coefficients get shrunk to zero, the output would equal the intercept and be constant. Furthermore, I noticed that when leaving Leave one out cross validation, I get correlation -1. – M.Zatzenbreck Apr 2 at 11:12