# Tuning Order XGBoost

A random forest in XGBoost has a lot of hyperparameters to tune. I have seen examples where people search over a handful of parameters at a time and others where they search over all of them simultaneously. What are some approaches for tuning the XGBoost hyper-parameters? And what is the rational for these approaches?

• I think you question should be modified a little. "best" - there is not answer for that. ask something like "what are some good methods" to do tuning?
– user98374
Commented Feb 15, 2017 at 6:02

Here is a good article on the topic:

Complete Guide to Parameter Tuning in XGBoost (with codes in Python)

Also, some people have had good success using hyperopt for tuning hyperparameters. Amine Benhalloum provides some Python code for tuning XGBoost: https://github.com/bamine/Kaggle-stuff/tree/master/otto

param_grid = {
'silent': [1],
'max_depth': [4,5,6,7],
'learning_rate': [0.001, 0.01, 0.1, 0.2, 0,3],
'subsample': [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'colsample_bytree': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'colsample_bylevel': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
'min_child_weight': [0.5, 1.0, 3.0, 5.0, 7.0, 10.0],
'gamma': [0, 0.25, 0.5, 1.0],
'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
'n_estimators': [100]}
fit_params = {'eval_metric': 'logloss',
'early_stopping_rounds': 10,
'eval_set': [(X_train_tfidf, y_train_tfidf)],
'verbose' : False
}

clf = xgb.XGBClassifier(n_jobs=-1)
randomized_search = RandomizedSearchCV(clf, param_grid, n_iter=30,
n_jobs=-1, verbose=0, cv=5,
fit_params=fit_params,
scoring='neg_log_loss', refit=False, random_state=42)
randomized_search.fit(X_train, y_train)