How to evaluate whether model is overfitting or underfitting when using cross_val_score and GridSearchCV?

Question

This is something that has been written about extensively, but I'm just confused about a couple of particular things which I haven't found a clear explanation of.

When cross validation is not used, data can be split into train and test, and trained on the train set. The model can then be evaluated on both sets and the goal is to have similar performance on either set, which means neither over/under-fit.

As far as I understand, when cross-validation is used, this removes the need to split into train and test sets, since CV effectively performs this split a number of times (defined by the number of folds). However, averaging scores you get from cross validation returns just a single score. Should this be interpreted as the train or the test score from the previous case? or neither? How can we tell if the model is overfit or underfit?

I am wondering how this fits in with GridSearchCV, since I have read that you are supposed to split your data into a train and validation set to confirm that your performance metric remains approximately the same. Is this necessary since we can just assume the model is not over/under-fit since we allow GridSearchCV to choose the best hyperparameters?

Furthermore, I have read something confusing in "Introduction to Machine Learning with Python" which says that data should be split into 3: train, val and test. The model is trained on the training set, and evaluated on the validation set in order to choose the best hyperparameters, and then taking the best hyperparameters is trained on train+val, and evaluated on test.

Answer 1

You need to check the accuracy difference between train and test set for each fold result. If your model gives you high training accuracy but low test accuracy so your model is overfitting. If your model does not give good training accuracy you can say your model is underfitting.

GridSearchCV is trying to find the best hyperparameters for your model. To do this, it splits the dataset into three-part. It uses a train set for the training part then test your data with validation set and tuning your parameters based on the validation set results. Finally, it uses test set to take the final model accuracy.

from sklearn.model_selection import KFold

kf = KFold(n_splits=5,random_state=42,shuffle=True)


# these are you training data points:
# features and targets
X = ....
y = ....

accuracies = []

for train_index, test_index in kf.split(X):

    data_train   = X[train_index]
    target_train = y[train_index]

    data_test    = X[test_index]
    target_test  = y[test_index]

    # if needed, do preprocessing here

    clf = LogisticRegression()
    clf.fit(data_train,target_train)

    test_preds = clf.predict(data_test)
    test_accuracy = accuracy_score(target_test,test_preds)

    train_preds = clf.predict(data_train)
    train_accuracy = accuracy_score(target_train, train_preds)

    print(train_accuracy, test_accuracy, (train_accuracy - test_accuracy) )

    accuracies.append(accuracy)

# this is the average accuracy over all folds
average_accuracy = np.mean(accuracies)

Answer 2

I am trying to answer each of the questions, you asked.

....... However, averaging scores you get from cross validation returns just a single score. Should this be interpreted as the train or the test score from the previous case? or neither?

I am not sure which library or package, you are using for cross-validation. I am assuming that you are using cross_val_score method (as it is widely used in tutorials). This method splits training set into k folds. Training on k-1 folds, the remaining fold is used as a test set to compute a performance. Thus, in a sense that you are not doing cross-validation for model selection, average accuracy you get from cross-validation is a estimate of test accuracy and you can call this test-score, not training score.

..... score from the previous case? or neither? How can we tell if the model is overfit or underfit?

I think your question of model overfit is regarding cross validation. You can easily understand whether your model is overfitted or not by comparing testing and training accuracy. However, for each of the k folds, cross_val_score gives you testing accuracy, not training accuracy. Hence, you should use sklearn's cross_validate which returns a dict containing test-score and others. And if you want to get training score as well, you just have to set value of return_train_score parameter to True.

A code snippet is following:

scores = cross_validate(rdn_forest_clf, train_features, train_lables, cv=5, scoring='f1', return_train_score=True)
print(scores.keys())  # Will print dict_keys(['fit_time', 'score_time', 'test_score', 'train_score'])
print(scores["train_score"])  # Will print your training score
print(scores["test_score"])  # Will print test score

.........confirm that your performance metric remains approximately the same. Is this necessary since we can just assume the model is not over/under-fit since we allow GridSearchCV to choose the best hyper-parameters?

Cross validation over the grid of hyper-parameters does not guarantee you about overfitting. Thus, to check whether the model you find by GridSearchCV is overfitted or not, you can use cv_results_ attribute of GridSearchCV. cv_results_ is a dictionary which contains details (e.g. mean_test_score, mean_score_time etc. ) for each combination of the parameters, given in parameters' grid. And to get training score related values (e.g. mean_train_score, std_train_score etc.), you have to pas return_train_score = True which is by default false.

Here is a code snipped to get mean training and testing accuracy for each combination of the parameters.

param_grid = {'n_estimators': [4, 5, 10, 15, 20, 30, 40, 60, 190, 500, 800], 'max_depth': [3, 4, 5, 6]}
grid_search_m = GridSearchCV(clf, param_grid, cv=5, scoring='f1', return_train_score=True)
grid_search_m.fit(train_features, train_lables)
print(grid_search_m.cv_results_.keys())
print(grid_search_m.cv_results_["mean_train_score"].shape)  # n_estimators: 11 values, max_depth: 4 values. Thus shape, 11*4=44
print(grid_search_m.cv_results_["mean_test_score"].shape)

print(grid_search_m.cv_results_["mean_train_score"])
print(grid_search_m.cv_results_["mean_test_score"])

Then, comparing training and testing accuracy, you can ensure whether your model is overfitted or not. You can check different SE questions also to find the strategies for this comparison.

Furthermore, I have read something confusing ...... model is trained on the training set, and evaluated on the validation set in order to choose the best hyper-parameters, and then taking the best hyper-parameters is trained on train+val, and evaluated on test.

This also can be done during working on a ML model, that is, using validation set instead of cross-validation. However, to avoid “wasting” too much training data in validation sets, a common technique is to use cross-validation.