I am trying to answer each of the questions, you asked.
....... However, averaging scores you get from cross validation returns just a single score. Should this be interpreted as the train or the test score from the previous case? or neither?
I am not sure which library or package, you are using for cross-validation. I am assuming that you are using cross_val_score method (as it is widely used in tutorials). This method splits training set into k folds. Training on k-1 folds, the remaining fold is used as a test set to compute a performance. Thus, in a sense that you are not doing cross-validation for model selection, average accuracy you get from cross-validation is a estimate of test accuracy and you can call this test-score, not training score.
..... score from the previous case? or neither? How can we tell if the model is overfit or underfit?
I think your question of model overfit is regarding cross validation. You can easily understand whether your model is overfitted or not by comparing testing and training accuracy. However, for each of the k folds, cross_val_score
gives you testing accuracy, not training accuracy. Hence, you should use sklearn's cross_validate which returns a dict containing test-score and others. And if you want to get training score as well, you just have to set value of return_train_score
parameter to True
.
A code snippet is following:
scores = cross_validate(rdn_forest_clf, train_features, train_lables, cv=5, scoring='f1', return_train_score=True)
print(scores.keys()) # Will print dict_keys(['fit_time', 'score_time', 'test_score', 'train_score'])
print(scores["train_score"]) # Will print your training score
print(scores["test_score"]) # Will print test score
.........confirm that your performance metric remains approximately the same. Is this necessary since we can just assume the model is not over/under-fit since we allow GridSearchCV to choose the best hyper-parameters?
Cross validation over the grid of hyper-parameters does not guarantee you about overfitting. Thus, to check whether the model you find by GridSearchCV is overfitted or not, you can use cv_results_ attribute of GridSearchCV
. cv_results_
is a dictionary which contains details (e.g. mean_test_score, mean_score_time etc. ) for each combination of the parameters, given in parameters' grid. And to get training score related values (e.g. mean_train_score, std_train_score etc.), you have to pas return_train_score = True
which is by default false.
Here is a code snipped to get mean training and testing accuracy for each combination of the parameters.
param_grid = {'n_estimators': [4, 5, 10, 15, 20, 30, 40, 60, 190, 500, 800], 'max_depth': [3, 4, 5, 6]}
grid_search_m = GridSearchCV(clf, param_grid, cv=5, scoring='f1', return_train_score=True)
grid_search_m.fit(train_features, train_lables)
print(grid_search_m.cv_results_.keys())
print(grid_search_m.cv_results_["mean_train_score"].shape) # n_estimators: 11 values, max_depth: 4 values. Thus shape, 11*4=44
print(grid_search_m.cv_results_["mean_test_score"].shape)
print(grid_search_m.cv_results_["mean_train_score"])
print(grid_search_m.cv_results_["mean_test_score"])
Then, comparing training and testing accuracy, you can ensure whether your model is overfitted or not. You can check different SE questions also to find the strategies for this comparison.
Furthermore, I have read something confusing ...... model is trained on the training set, and evaluated on the validation set in order to choose the best hyper-parameters, and then taking the best hyper-parameters is trained on train+val, and evaluated on test.
This also can be done during working on a ML model, that is, using validation set instead of cross-validation. However, to avoid “wasting” too much training data in validation sets, a common technique is to use cross-validation.