1
$\begingroup$

I am using 3-fold cross validation for hyperparameter selection of my XGBOOST model. To be specific, I use xgboost.cv for cross validation instead of sklearn. I use random search for hyperparameter search and choose the one or a few set of hyperparameters with the best average score in the hold-out fold of data. The standard deviation of score is approximately the same across all hyperparameter set.

What I observe is that the model is overfitting to the cross validation data. The score of model on testing data is always much worse than the average score on hold-out fold of data. So instead of selecting the set of hyperparameter with the best average score, is there any other criterion I should consider in order to reduce the overfitting?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ It can happen due to many reasons: 1) your model is too complex to be learned properly given the amount of data you have. 2) domain shift between your training and test data. 3) cross-validation is typically used when you don't have enough labelled data, so you concat everything you have and report cross-validated result instead of accuracy on test set. If you have enough data, just use 90% of your training data to train, 10% of it to tune the parameters and then apply it to your test data. $\endgroup$
    – user3639557
    May 11, 2018 at 15:43
  • 1
    $\begingroup$ Do you use separate sets for training, finding hyperparameters, & testing? If not, it makes overfitting even more likely. $\endgroup$
    – Tim ♦
    Oct 13, 2020 at 7:21

1 Answer 1

Reset to default
1
$\begingroup$

The average score of your agent on the validation sets in cross validation should not be significantly better than the score on the holdout (final test) set, because for each cross-validation fold the agent is as blind to the data in the validation set as your final model (presumably retrained on all the training data) is to the holdout data.

Is there some quality of your holdout set that is causing your results to be skewed? Are the examples in the training set easier somehow? If you select a different random holdout set, does the problem persist?

So long as the variance in performance across the folds is low, then so long as the holdout set is statistically similar to the training set, the performance should not degrade. Best practice is to select the parameters that yield the reliably (low variance across folds) best-performing (high mean) cross validation score. I am not aware of any alternative criteria nor arguments why one might prefer them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.