Tuning Order XGBoost

Question

A random forest in XGBoost has a lot of hyperparameters to tune. I have seen examples where people search over a handful of parameters at a time and others where they search over all of them simultaneously. What are some approaches for tuning the XGBoost hyper-parameters? And what is the rational for these approaches?

Answer 1

Here is a good article on the topic:

Complete Guide to Parameter Tuning in XGBoost (with codes in Python)

Also, some people have had good success using hyperopt for tuning hyperparameters. Amine Benhalloum provides some Python code for tuning XGBoost: https://github.com/bamine/Kaggle-stuff/tree/master/otto

Answer 2

param_grid = {
    'silent': [1],
    'max_depth': [4,5,6,7],
    'learning_rate': [0.001, 0.01, 0.1, 0.2, 0,3],
    'subsample': [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'colsample_bytree': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'colsample_bylevel': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'min_child_weight': [0.5, 1.0, 3.0, 5.0, 7.0, 10.0],
    'gamma': [0, 0.25, 0.5, 1.0],
    'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
    'n_estimators': [100]}
    fit_params = {'eval_metric': 'logloss',
          'early_stopping_rounds': 10,
          'eval_set': [(X_train_tfidf, y_train_tfidf)],
          'verbose' : False
          }

    clf = xgb.XGBClassifier(n_jobs=-1)
    randomized_search = RandomizedSearchCV(clf, param_grid, n_iter=30,
                        n_jobs=-1, verbose=0, cv=5,
                        fit_params=fit_params,
                        scoring='neg_log_loss', refit=False, random_state=42)
    randomized_search.fit(X_train, y_train)