# Large Negative r-Squared Scores using Cross-Validation

I am working working with the World Happiness Report dataset from Kaggle. When using either cross_val_score or GridSearchCV from sklearn, I get very large negative r2 scores. My first thought was that the models I was using were SEVERELY over-fitting (it is a small dataset), but when I performed cross-validation using KFold to split the data, I got reasonable results.

You can view an example of what I am talking about in this Google Colab Notebook. The relevant code is also shown below.

Using cross_val_score

model = LinearRegression()
print(cross_val_score(model, X, y, scoring='r2', cv=5))


Output: [-5.57285067 -5.9477523 -6.23988074 -8.84930385 -2.39521998]

Using KFold

model = LinearRegression()
kf = KFold(n_splits=5, random_state=1, shuffle=True)
scores = []

for i, (train_index, test_index) in enumerate(kf.split(X)):
X_train = X[train_index,:]
y_train = y[train_index]
X_test = X[test_index,:]
y_test = y[test_index]

model.fit(X_train, y_train)
test_score = model.score(X_test, y_test)
scores.append(round(test_score, 6))

print(scores)


Output: [0.829785, 0.774577, 0.762708, 0.661945, 0.727391]

• It doesn't seem to matter what type of model I use. I still get very large negative scores when using cross_val_score.
• I created a synthetic dataset that was approximately the same size of the World Happiness dataset just to try some things out. In that case, I did not get large negative r2 scores from cross_val_score. This is shown in the Google Colab notebook that I shared above.
• I notice that the magnitude of the negative results I get using cross_val_score is greatly affected by the number of folds I use. Increasing the number of folds significantly increases the magnitude.

• Thank you very much for your reply, but I don't think that it quite gets to the core of the issue. First, the primary purpose of cross-validation is to avoid needing to create a train-test split. New splits are created during cross-validation. Second, you are not providing a "train_test_split" object to cross_valscore. The objects X_train and y_train are numpy arrays, just like the original X and y arrays. – Beane Mar 7 at 17:00
• Your comment did help me to discover the true source of the issue, however. The data is not shuffled by cross_val_score. I did not know this previously. Since the observations are provided in order of the target variable in this dataset, the folds created by cross-val_score each consider of similar observations and are not representative of the dataset as a whole. I was able to resolve the issue by shuffling the data before using cross_val_score. Thank you for taking the time to look at this! – Beane Mar 7 at 17:00
• Please try not to take things so personally. I appreciate your effort to help, and was not trying to be critical of you, but the points I made were correct. In reference to your comment about arrays vs DataFrames... If you pass arrays to train_test_split, you will get back out arrays. If you pass in DataFrames, you will get back DataFrames. I said that X_train was an array because in my case, it was. I has created X to be an array, meaning that X_train was an array. (continued) – Beane Mar 7 at 21:13
• But if I has been working with DataFrames, my point still stands: The datatype of X and X_train will be the same (both arrays or both DataFrames). You suggested that cross_val_score expected a train_test_split object, but that is not the issue. The type of the objects are the same with or without using train_test_split. As I explained, the reason why train_test_split works in this situation is because it shuffles the rows before performing the split. – Beane Mar 7 at 21:16
Solved! The issue is that the observations in this dataset are ordered with respect to the value of the target variable and cross_val_score` does not shuffle the data provided to it. As a result, each fold contains a set of values that are similar to each other and don't represent the entire dataset.