5
$\begingroup$

This is something that has been written about extensively, but I'm just confused about a couple of particular things which I haven't found a clear explanation of.

When cross validation is not used, data can be split into train and test, and trained on the train set. The model can then be evaluated on both sets and the goal is to have similar performance on either set, which means neither over/under-fit.

As far as I understand, when cross-validation is used, this removes the need to split into train and test sets, since CV effectively performs this split a number of times (defined by the number of folds). However, averaging scores you get from cross validation returns just a single score. Should this be interpreted as the train or the test score from the previous case? or neither? How can we tell if the model is overfit or underfit?

I am wondering how this fits in with GridSearchCV, since I have read that you are supposed to split your data into a train and validation set to confirm that your performance metric remains approximately the same. Is this necessary since we can just assume the model is not over/under-fit since we allow GridSearchCV to choose the best hyperparameters?

Furthermore, I have read something confusing in "Introduction to Machine Learning with Python" which says that data should be split into 3: train, val and test. The model is trained on the training set, and evaluated on the validation set in order to choose the best hyperparameters, and then taking the best hyperparameters is trained on train+val, and evaluated on test.

$\endgroup$

2 Answers 2

6
$\begingroup$

You need to check the accuracy difference between train and test set for each fold result. If your model gives you high training accuracy but low test accuracy so your model is overfitting. If your model does not give good training accuracy you can say your model is underfitting.

GridSearchCV is trying to find the best hyperparameters for your model. To do this, it splits the dataset into three-part. It uses a train set for the training part then test your data with validation set and tuning your parameters based on the validation set results. Finally, it uses test set to take the final model accuracy.

from sklearn.model_selection import KFold

kf = KFold(n_splits=5,random_state=42,shuffle=True)


# these are you training data points:
# features and targets
X = ....
y = ....

accuracies = []

for train_index, test_index in kf.split(X):

    data_train   = X[train_index]
    target_train = y[train_index]

    data_test    = X[test_index]
    target_test  = y[test_index]

    # if needed, do preprocessing here

    clf = LogisticRegression()
    clf.fit(data_train,target_train)

    test_preds = clf.predict(data_test)
    test_accuracy = accuracy_score(target_test,test_preds)

    train_preds = clf.predict(data_train)
    train_accuracy = accuracy_score(target_train, train_preds)

    print(train_accuracy, test_accuracy, (train_accuracy - test_accuracy) )

    accuracies.append(accuracy)

# this is the average accuracy over all folds
average_accuracy = np.mean(accuracies)
$\endgroup$
6
  • $\begingroup$ Regarding your first paragraph, how do I go about finding the difference in accuracy between train and test set for each fold? cross_val_score just returns one score, which one is this? And about GridSearchCV, I hadn't realised that it does everything for you basically. Thanks for the help! $\endgroup$ Dec 5, 2019 at 17:23
  • $\begingroup$ You can use "cross_val_score" or "cross_validate" functions from scikit-learn to check accuracy. (cross_val_score(model, X, y, cv=5)) will give you 5 results. $\endgroup$
    – Batuhan B
    Dec 5, 2019 at 17:29
  • $\begingroup$ Sorry to sound so blunt, but how do I find the difference between train and test set performance based on the 5 results from cross_val_score as you stated? $\endgroup$ Dec 5, 2019 at 17:32
  • $\begingroup$ @TimofeyAbramski I edited my answer and added a code snippet. You can use KFold method. $\endgroup$
    – Batuhan B
    Dec 5, 2019 at 17:51
  • $\begingroup$ Thank you very much for taking the time to write up this code. I am wondering, is there any way to automate this through cross_val_score? And also, the scores which cross_val_score returns, is that the train_accuracy or test_accuracy according to your code? Thanks again, this will come in handy $\endgroup$ Dec 5, 2019 at 18:37
1
$\begingroup$

I am trying to answer each of the questions, you asked.

....... However, averaging scores you get from cross validation returns just a single score. Should this be interpreted as the train or the test score from the previous case? or neither?

I am not sure which library or package, you are using for cross-validation. I am assuming that you are using cross_val_score method (as it is widely used in tutorials). This method splits training set into k folds. Training on k-1 folds, the remaining fold is used as a test set to compute a performance. Thus, in a sense that you are not doing cross-validation for model selection, average accuracy you get from cross-validation is a estimate of test accuracy and you can call this test-score, not training score.

..... score from the previous case? or neither? How can we tell if the model is overfit or underfit?

I think your question of model overfit is regarding cross validation. You can easily understand whether your model is overfitted or not by comparing testing and training accuracy. However, for each of the k folds, cross_val_score gives you testing accuracy, not training accuracy. Hence, you should use sklearn's cross_validate which returns a dict containing test-score and others. And if you want to get training score as well, you just have to set value of return_train_score parameter to True.

A code snippet is following:

scores = cross_validate(rdn_forest_clf, train_features, train_lables, cv=5, scoring='f1', return_train_score=True)
print(scores.keys())  # Will print dict_keys(['fit_time', 'score_time', 'test_score', 'train_score'])
print(scores["train_score"])  # Will print your training score
print(scores["test_score"])  # Will print test score

.........confirm that your performance metric remains approximately the same. Is this necessary since we can just assume the model is not over/under-fit since we allow GridSearchCV to choose the best hyper-parameters?

Cross validation over the grid of hyper-parameters does not guarantee you about overfitting. Thus, to check whether the model you find by GridSearchCV is overfitted or not, you can use cv_results_ attribute of GridSearchCV. cv_results_ is a dictionary which contains details (e.g. mean_test_score, mean_score_time etc. ) for each combination of the parameters, given in parameters' grid. And to get training score related values (e.g. mean_train_score, std_train_score etc.), you have to pas return_train_score = True which is by default false.

Here is a code snipped to get mean training and testing accuracy for each combination of the parameters.

param_grid = {'n_estimators': [4, 5, 10, 15, 20, 30, 40, 60, 190, 500, 800], 'max_depth': [3, 4, 5, 6]}
grid_search_m = GridSearchCV(clf, param_grid, cv=5, scoring='f1', return_train_score=True)
grid_search_m.fit(train_features, train_lables)
print(grid_search_m.cv_results_.keys())
print(grid_search_m.cv_results_["mean_train_score"].shape)  # n_estimators: 11 values, max_depth: 4 values. Thus shape, 11*4=44
print(grid_search_m.cv_results_["mean_test_score"].shape)

print(grid_search_m.cv_results_["mean_train_score"])
print(grid_search_m.cv_results_["mean_test_score"])

Then, comparing training and testing accuracy, you can ensure whether your model is overfitted or not. You can check different SE questions also to find the strategies for this comparison.

Furthermore, I have read something confusing ...... model is trained on the training set, and evaluated on the validation set in order to choose the best hyper-parameters, and then taking the best hyper-parameters is trained on train+val, and evaluated on test.

This also can be done during working on a ML model, that is, using validation set instead of cross-validation. However, to avoid “wasting” too much training data in validation sets, a common technique is to use cross-validation.

$\endgroup$
1
  • $\begingroup$ If I first test several models with their standard parameters using Cross-Validate, and choose the one with the least error and least variation, and apply GridSearchCV to that model, it would be guaranteed that the resulting model of GridSearchCV will not suffer from overfitting or underfitting or imbalance of lessons? I ask this because every time the authors mention GridSearchCV, they don't mention anything else, as if nothing else needs to be done (model evaluation metrics, overfitting/underfitting test). I would appreciate your response. $\endgroup$
    – user140259
    Mar 15, 2022 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.