6
$\begingroup$

A random forest in XGBoost has a lot of hyperparameters to tune. I have seen examples where people search over a handful of parameters at a time and others where they search over all of them simultaneously. What are some approaches for tuning the XGBoost hyper-parameters? And what is the rational for these approaches?

$\endgroup$
1
  • $\begingroup$ I think you question should be modified a little. "best" - there is not answer for that. ask something like "what are some good methods" to do tuning? $\endgroup$
    – user98374
    Feb 15, 2017 at 6:02

2 Answers 2

2
$\begingroup$

Here is a good article on the topic:

Complete Guide to Parameter Tuning in XGBoost (with codes in Python)

Also, some people have had good success using hyperopt for tuning hyperparameters. Amine Benhalloum provides some Python code for tuning XGBoost: https://github.com/bamine/Kaggle-stuff/tree/master/otto

$\endgroup$
2
$\begingroup$
param_grid = {
    'silent': [1],
    'max_depth': [4,5,6,7],
    'learning_rate': [0.001, 0.01, 0.1, 0.2, 0,3],
    'subsample': [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'colsample_bytree': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'colsample_bylevel': [0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    'min_child_weight': [0.5, 1.0, 3.0, 5.0, 7.0, 10.0],
    'gamma': [0, 0.25, 0.5, 1.0],
    'reg_lambda': [0.1, 1.0, 5.0, 10.0, 50.0, 100.0],
    'n_estimators': [100]}
    fit_params = {'eval_metric': 'logloss',
          'early_stopping_rounds': 10,
          'eval_set': [(X_train_tfidf, y_train_tfidf)],
          'verbose' : False
          }

    clf = xgb.XGBClassifier(n_jobs=-1)
    randomized_search = RandomizedSearchCV(clf, param_grid, n_iter=30,
                        n_jobs=-1, verbose=0, cv=5,
                        fit_params=fit_params,
                        scoring='neg_log_loss', refit=False, random_state=42)
    randomized_search.fit(X_train, y_train)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.