What is the correct procedure to optimize hyper parameters?

I'm doing a 10 fold cross validation to find the best performing xgboost model. I'm then taking the model that performed best and using random search or grid search to optimize the hyper parameters nrounds, eta, gamma, min child weight, and max depth. The results of the optimization always show the original hyper parameters are best. I must be biasing the model to the original hyper parameters through the 10 fold cross validation. Am I supposed to do two separate cross validations? One for hyper parameters and one for model parameters?


1 Answer 1


In general:

When you do a cross validation with 10 fold, you are actually making this folds for hyper parameter optimization already. Imagine you have a 5 fold instead.

4 folds are train (pink), 1 fold is the validation set (or test set of your fold) (blue) for your hyperparameters. If your hyperparameters are optimized you can then test this hyperparameter combination on your test set (violet), with a predict method.

enter image description here

When you use GridSearch or RandomizedSearch there is already a cv option inside: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

The GridSearchCV already has CV in its name and a cv option where you can insert an int for the amount of folds you wanna do (there is also a default option).

The Grid and RadnomizedSearchCV are a convenient way of not doing an extra fold command before predicting the performance of your model on the test data.

So to answer your question, you do the cv only once, within the GridsearchCV command (or RandomizedCV, whatever you prefer):

             param_grid={'gamma': [1, 10], ...)}), cv=10)

Besides, if you mean model parameters are coefficients, they will be automatically trained by the method, the hyperparameters will be trained by the fold. So basically, you do this two steps as you described it, at once in the kfolds. Because of that you only need one fold method.

  • $\begingroup$ Thank you for the reply. It sounds like what is happening is gridsearchCV and randomsearchCV is doing the hyper parameter turning and model parameter turning at the same time? If there are 3 different models and 3 different sets of hyper parameters the first fold would run 9 times? That seems like it could take forever to complete. Quantum computing can't get here soon enough. $\endgroup$
    – Eric
    Commented May 11, 2021 at 17:48
  • $\begingroup$ GridSearch really looks for the best solution(so yeah its exhaustive) randomized just gives a random result after a few tries, but experience shows that randomized is way way faster and still helps you in getting good results. Use only Grid Search if you have a borrowed computational power in terms of cpu cores and gpus, when you only want to sqeeze out a few %. But before that randomized is always ok. $\endgroup$ Commented May 11, 2021 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.