Hyperparameter tuning using grid search/randomised search

Question

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn GridSearchCV and RandomizedSearchCV functions.

The question is, should I tune all the model parameters at once or sequentally in batches, i.e., tune learning_rate and n_estimators first, then fix these and do the same for the rest?

Parameters for GridSearchCV:

param =     {
            'max_depth': [2, 3, 4, 5],
            'learning_rate': [0.05, 0.1, 0.15, 0.2, 0.25, 0.3],
            'n_estimators': [50, 100, 150, 200],
            'gamma': [0, 0.1, 0.2],
            'min_child_weight': [0, 0.2, 0.4, 0.6, 0.8, 1],
            'max_delta_step': [0],
            'subsample': [0.7, 0.8, 0.9, 1],
            'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9, 1],
            'colsample_bylevel': [1],
            'reg_alpha': [0, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2],
            'reg_lambda': [0, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2],
            'base_score': [0.5]
            }

Parameters for RandomizedSearchCV:

import scipy as sp

param =     {
            'max_depth': sp.stats.randint(2, 6),
            'learning_rate': sp.stats.uniform(0.05, 0.25),
            'n_estimators': sp.stats.randint(30, 201),
            'gamma': sp.stats.uniform(0, 0.2),
            'min_child_weight': sp.stats.uniform(0, 1),
            'max_delta_step': [0],
            'subsample': sp.stats.uniform(0.7, 0.3),
            'colsample_bytree': sp.stats.uniform(0.5, 0.5),
            'colsample_bylevel': [1],
            'reg_alpha': sp.stats.reciprocal(1e-5, 1e2),
            'reg_lambda': sp.stats.reciprocal(1e-5, 1e2),
            'base_score': [0.5]
            }

Answer 1

Just saying, if your tests are cheap, nothing is better than trying them all. But, this is not the case in general, and here I assume the same. And, just be aware that tuning a subset of parameters and then focusing others is not the same as trying all (all you described actually) options. The former method sacrifices some possibilities to be faster, but it's not a bad heuristic. Of course, while searching for your parameters, it's your responsibility to fix others to reasonable values.

What you can do alternatively is to apply your grid search at a finer granularity where your algorithm performs best, based on your previous analyses and apply this approach repeatedly. Or you can explore some random number of combinations via RandomizedSearchCV and again do local search around the best options.

What I'd also suggest is to use BayesSearchCV in scikit-optimize package, which automatically chooses the next combination to try using previously computed performances. Based on the output of this, you might again apply a more granular local search around the best candidates.