I am conducting hyperparameter tuning for my XGBClassifier
model for a multi-class classification problem using scikit-learn GridSearchCV
and RandomizedSearchCV
functions.
The question is, should I tune all the model parameters at once or sequentally in batches, i.e., tune learning_rate
and n_estimators
first, then fix these and do the same for the rest?
Parameters for GridSearchCV
:
param = {
'max_depth': [2, 3, 4, 5],
'learning_rate': [0.05, 0.1, 0.15, 0.2, 0.25, 0.3],
'n_estimators': [50, 100, 150, 200],
'gamma': [0, 0.1, 0.2],
'min_child_weight': [0, 0.2, 0.4, 0.6, 0.8, 1],
'max_delta_step': [0],
'subsample': [0.7, 0.8, 0.9, 1],
'colsample_bytree': [0.5, 0.6, 0.7, 0.8, 0.9, 1],
'colsample_bylevel': [1],
'reg_alpha': [0, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2],
'reg_lambda': [0, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1, 1e1, 1e2],
'base_score': [0.5]
}
Parameters for RandomizedSearchCV
:
import scipy as sp
param = {
'max_depth': sp.stats.randint(2, 6),
'learning_rate': sp.stats.uniform(0.05, 0.25),
'n_estimators': sp.stats.randint(30, 201),
'gamma': sp.stats.uniform(0, 0.2),
'min_child_weight': sp.stats.uniform(0, 1),
'max_delta_step': [0],
'subsample': sp.stats.uniform(0.7, 0.3),
'colsample_bytree': sp.stats.uniform(0.5, 0.5),
'colsample_bylevel': [1],
'reg_alpha': sp.stats.reciprocal(1e-5, 1e2),
'reg_lambda': sp.stats.reciprocal(1e-5, 1e2),
'base_score': [0.5]
}