I observed than when using Lasso regression and KFold crossvalidation with my data, predicted values show negative correlation with observed values.
I tried to replicate the problem with a randomly generated dataset roughly similar to the one I am working on (10 features, 5 of which are correlated with the response variable and 500 observations) and see something similar.
#Lasso example from sklearn.linear_model import Lasso from sklearn.model_selection import KFold from scipy.stats import pearsonr import numpy as np #create random features rng = np.random.RandomState(seed=42) X = rng.randn(2500).reshape(500, 5) #create additional correlated random features #create covariance matrix covs = np.full((6,6), 0.8) #set correlation (== covariance since means are 0 and sd is one) to 0.8 for all of these variables np.fill_diagonal(covs, 1) means = np.zeros(6) X_correlated = rng.multivariate_normal(means, covs, 500) #append all of the correlated features but one to X X = np.c_[X, X_correlated[:,:-1]] #set the last of the correlated features as Y Y = X_correlated[:,-1] #instantiate regressor and cv object cv = KFold(100) reg = Lasso(random_state=42) #create arrays to save predicted (and observed) Y values pred = np.array() obs = np.array() #run cross validation for train, test in cv.split(X): #fit regressor reg.fit(X[train], Y[train]) #append predicted and observed values to the arrays pred = np.r_[pred, reg.predict(X[test])] obs = np.r_[obs, Y[test]] #test correlation pearsonr(pred, obs) #r = -0.424, p < 10**(-22)
In my hands, using L2 penalized regression (
LinearRegression() in the same setting produces results that make sense. Does anyone have any idea what I am doing wrong?