0
$\begingroup$

I am working working with the World Happiness Report dataset from Kaggle. When using either cross_val_score or GridSearchCV from sklearn, I get very large negative r2 scores. My first thought was that the models I was using were SEVERELY over-fitting (it is a small dataset), but when I performed cross-validation using KFold to split the data, I got reasonable results.

You can view an example of what I am talking about in this Google Colab Notebook. The relevant code is also shown below.

Using cross_val_score

model = LinearRegression()
print(cross_val_score(model, X, y, scoring='r2', cv=5))

Output: [-5.57285067 -5.9477523 -6.23988074 -8.84930385 -2.39521998]

Using KFold

model = LinearRegression()
kf = KFold(n_splits=5, random_state=1, shuffle=True)
scores = []

for i, (train_index, test_index) in enumerate(kf.split(X)):
    X_train = X[train_index,:]
    y_train = y[train_index]
    X_test = X[test_index,:]
    y_test = y[test_index]

    model.fit(X_train, y_train)
    test_score = model.score(X_test, y_test)
    scores.append(round(test_score, 6))

print(scores)

Output: [0.829785, 0.774577, 0.762708, 0.661945, 0.727391]

Some Additional Observations

  • It doesn't seem to matter what type of model I use. I still get very large negative scores when using cross_val_score.
  • I created a synthetic dataset that was approximately the same size of the World Happiness dataset just to try some things out. In that case, I did not get large negative r2 scores from cross_val_score. This is shown in the Google Colab notebook that I shared above.
  • I notice that the magnitude of the negative results I get using cross_val_score is greatly affected by the number of folds I use. Increasing the number of folds significantly increases the magnitude.

Thanks in advance for your help!

$\endgroup$
1
$\begingroup$

you are fitting your whole data to it, it seems the cross_val_score awaits a predefined train_test_split object. Ignore the part with 'bla' :-). Btw. it is also the kaggle data. I used it in one of my own notebooks.

Solve

$\endgroup$
6
  • $\begingroup$ Thank you very much for your reply, but I don't think that it quite gets to the core of the issue. First, the primary purpose of cross-validation is to avoid needing to create a train-test split. New splits are created during cross-validation. Second, you are not providing a "train_test_split" object to cross_valscore. The objects X_train` and y_train are numpy arrays, just like the original X and y arrays. $\endgroup$ – Beane Mar 7 at 17:00
  • $\begingroup$ Your comment did help me to discover the true source of the issue, however. The data is not shuffled by cross_val_score. I did not know this previously. Since the observations are provided in order of the target variable in this dataset, the folds created by cross-val_score each consider of similar observations and are not representative of the dataset as a whole. I was able to resolve the issue by shuffling the data before using cross_val_score. Thank you for taking the time to look at this! $\endgroup$ – Beane Mar 7 at 17:00
  • $\begingroup$ Well if you see it that way: Xtrain is not a numpy array it is pandas DataFrame object... which is an array like shape. But I'm glad I could help even you tried to correct me in every sentence ... $\endgroup$ – Patrick Bormann Mar 7 at 17:44
  • $\begingroup$ Please try not to take things so personally. I appreciate your effort to help, and was not trying to be critical of you, but the points I made were correct. In reference to your comment about arrays vs DataFrames... If you pass arrays to train_test_split, you will get back out arrays. If you pass in DataFrames, you will get back DataFrames. I said that X_train was an array because in my case, it was. I has created X to be an array, meaning that X_train was an array. (continued) $\endgroup$ – Beane Mar 7 at 21:13
  • $\begingroup$ But if I has been working with DataFrames, my point still stands: The datatype of X and X_train will be the same (both arrays or both DataFrames). You suggested that cross_val_score expected a train_test_split object, but that is not the issue. The type of the objects are the same with or without using train_test_split. As I explained, the reason why train_test_split works in this situation is because it shuffles the rows before performing the split. $\endgroup$ – Beane Mar 7 at 21:16
-1
$\begingroup$

Solved! The issue is that the observations in this dataset are ordered with respect to the value of the target variable and cross_val_score does not shuffle the data provided to it. As a result, each fold contains a set of values that are similar to each other and don't represent the entire dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.