# How to tune hyperparameters of xgboost trees?

I have a class imbalanced data & I want to tune the hyperparameters of the boosted tress using xgboost.

Questions

1. Is there an equivalent of gridsearchcv or randomsearchcv for xgboost?
2. If not what is the recommended approach to tune the parameters of xgboost?
• stats.stackexchange.com/questions/155883/… – Aakash Gupta Sep 6 '15 at 17:29
• Thanks but that link discusses a different issue & doesnt answer my question. – GeorgeOfTheRF Sep 12 '15 at 9:40
• Is the exact naming of the parameter xgboost(max.depth) or xgb.train(max_depth)? Does xgboost inconsistently use dot vs underscore for the parameter in different places? Or are they converted? – smci Mar 6 '17 at 5:51
• @smci , check "help("xgboost-deprecated")" – Hemant Rupani May 19 '17 at 11:55

Since the interface to xgboost in caret has recently changed, here is a script that provides a fully commented walkthrough of using caret to tune xgboost hyper-parameters.

For this, I will be using the training data from the Kaggle competition "Give Me Some Credit".

## 1. Fitting an xgboost model

In this section, we:

• fit an xgboost model with arbitrary hyperparameters
• evaluate the loss (AUC-ROC) using cross-validation (xgb.cv)
• plot the training versus testing evaluation metric

Here is some code to do this.

library(caret)
library(xgboost)
library(dplyr)
library(tidyr)

# load in the training data
na.omit() %>%                                                                # listwise deletion
select(-[EMPTY]) %>%
mutate(SeriousDlqin2yrs = factor(SeriousDlqin2yrs,                           # factor variable for classification
labels = c("Failure", "Success")))

# xgboost fitting with arbitrary parameters
xgb_params_1 = list(
objective = "binary:logistic",                                               # binary classification
eta = 0.01,                                                                  # learning rate
max.depth = 3,                                                               # max tree depth
eval_metric = "auc"                                                          # evaluation/loss metric
)

# fit the model with the arbitrary parameters specified above
xgb_1 = xgboost(data = as.matrix(df_train %>%
select(-SeriousDlqin2yrs)),
label = df_train$SeriousDlqin2yrs, params = xgb_params_1, nrounds = 100, # max number of trees to build verbose = TRUE, print.every.n = 1, early.stop.round = 10 # stop if no improvement within 10 trees ) # cross-validate xgboost to get the accurate measure of error xgb_cv_1 = xgb.cv(params = xgb_params_1, data = as.matrix(df_train %>% select(-SeriousDlqin2yrs)), label = df_train$SeriousDlqin2yrs,
nrounds = 100,
nfold = 5,                                                   # number of folds in K-fold
prediction = TRUE,                                           # return the prediction using the final model
showsd = TRUE,                                               # standard deviation of loss across folds
stratified = TRUE,                                           # sample is unbalanced; use stratified sampling
verbose = TRUE,
print.every.n = 1,
early.stop.round = 10
)

# plot the AUC for the training and testing samples
xgb_cv_1$dt %>% select(-contains("std")) %>% mutate(IterationNum = 1:n()) %>% gather(TestOrTrain, AUC, -IterationNum) %>% ggplot(aes(x = IterationNum, y = AUC, group = TestOrTrain, color = TestOrTrain)) + geom_line() + theme_bw()  Here is what the testing versus training AUC looks like: ## 2. Hyperparameter search using train For the hyperparameter search, we perform the following steps: • create a data.frame with unique combinations of parameters that we want trained models for. • Specify the control parameters that apply to each model's training, including the cross-validation parameters, and specify that the probabilities be computed so that the AUC can be computed • cross-validate & train the models for each parameter combination, saving the AUC for each model. Here is some code that shows how to do this. # set up the cross-validated hyper-parameter search xgb_grid_1 = expand.grid( nrounds = 1000, eta = c(0.01, 0.001, 0.0001), max_depth = c(2, 4, 6, 8, 10), gamma = 1 ) # pack the training control parameters xgb_trcontrol_1 = trainControl( method = "cv", number = 5, verboseIter = TRUE, returnData = FALSE, returnResamp = "all", # save losses across all models classProbs = TRUE, # set to TRUE for AUC to be computed summaryFunction = twoClassSummary, allowParallel = TRUE ) # train the model for each parameter combination in the grid, # using CV to evaluate xgb_train_1 = train( x = as.matrix(df_train %>% select(-SeriousDlqin2yrs)), y = as.factor(df_train$SeriousDlqin2yrs),
trControl = xgb_trcontrol_1,
tuneGrid = xgb_grid_1,
method = "xgbTree"
)

# scatter plot of the AUC against max_depth and eta
ggplot(xgb_train_1$results, aes(x = as.factor(eta), y = max_depth, size = ROC, color = ROC)) + geom_point() + theme_bw() + scale_size_continuous(guide = "none")  Lastly, you can create a bubbleplot for the AUC over the variations of eta and max_depth: • Does caret still only support eta, gamma and max depth for grid search what about subsample and other parameters of xgboost? – GeorgeOfTheRF Nov 13 '15 at 13:56 • @ML_Pro Support for most xgboost parameters now exists, in particular support for gamma is new. Here is a full list of supported parameters. – tchakravarty Nov 13 '15 at 14:03 • That is support by xgboost right? My question is about which all parameters does caret support for grid search – GeorgeOfTheRF Nov 13 '15 at 14:37 • What would be the changes required for multiclass classification. Also documentation says use scale_pose_weight for imbalanced classification. Can you provide details on how to? Thanks! – discipulus Feb 16 '16 at 7:31 • For the unbalanced class issue, scale_pos_weight is now documented in the parameter documentation. scale_pos_weight is not a caret tuning parameter, but you can compare manually. In my case, using the weight happened to have little effect (binary classification, >20% positives) – geneorama May 19 '16 at 20:33 Caret package have incorporated xgboost. cv.ctrl <- trainControl(method = "repeatedcv", repeats = 1,number = 3, #summaryFunction = twoClassSummary, classProbs = TRUE, allowParallel=T) xgb.grid <- expand.grid(nrounds = 1000, eta = c(0.01,0.05,0.1), max_depth = c(2,4,6,8,10,14) ) set.seed(45) xgb_tune <-train(formula, data=train, method="xgbTree", trControl=cv.ctrl, tuneGrid=xgb.grid, verbose=T, metric="Kappa", nthread =3 )  Sample output eXtreme Gradient Boosting 32218 samples 41 predictor 2 classes: 'N', 'Y' No pre-processing Resampling: Cross-Validated (3 fold, repeated 1 times) Summary of sample sizes: 21479, 21479, 21478 Resampling results Accuracy Kappa Accuracy SD Kappa SD 0.9324911 0.1094426 0.0009742774 0.008972911  One drawback i see is that other parameters of xgboost like subsample etc are not supported by caret currently. Edit Gamma, colsample_bytree, min_child_weight and subsample etc can now (June 2017) be tuned directly using Caret. Just add them in the grid portion of the above code to make it work. Thanks usεr11852 for highliting it in the comment. • A minor update regarding the drawback mentioned. caret now (Feb-2017) supports additional parameters for gamma, colsample_bytree, min_child_weight and subsample. (So effectively you can tune almost everything - given time) – usεr11852 says Reinstate Monic Feb 25 '17 at 15:09 I know this is an old question, but I use a different method from the ones above. I use the BayesianOptimization function from the Bayesian Optimization package to find optimal parameters. To do this, you first create cross validation folds, then create a function xgb.cv.bayes that has as parameters the boosting hyper parameters you want to change. In this example I am tuning max.depth, min_child_weight, subsample, colsample_bytree, gamma. You then call xgb.cv in that function with the hyper parameters set to in the input parameters of xgb.cv.bayes. Then you call BayesianOptimization with the xgb.cv.bayes and the desired ranges of the boosting hyper parameters. init_points is the number of initial models with hyper parameters taken randomly from the specified ranges, and n_iter is the number of rounds of models after the initial points. The function outputs all boosting parameters and the test AUC. cv_folds <- KFold(as.matrix(df.train[,target.var]), nfolds = 5, stratified = TRUE, seed = 50) xgb.cv.bayes <- function(max.depth, min_child_weight, subsample, colsample_bytree, gamma){ cv <- xgv.cv(params = list(booster = 'gbtree', eta = 0.05, max_depth = max.depth, min_child_weight = min_child_weight, subsample = subsample, colsample_bytree = colsample_bytree, gamma = gamma, lambda = 1, alpha = 0, objective = 'binary:logistic', eval_metric = 'auc'), data = data.matrix(df.train[,-target.var]), label = as.matrix(df.train[, target.var]), nround = 500, folds = cv_folds, prediction = TRUE, showsd = TRUE, early.stop.round = 5, maximize = TRUE, verbose = 0 ) list(Score = cv$dt[, max(test.auc.mean)],
Pred = cv$pred) } xgb.bayes.model <- BayesianOptimization( xgb.cv.bayes, bounds = list(max.depth = c(2L, 12L), min_child_weight = c(1L, 10L), subsample = c(0.5, 1), colsample_bytree = c(0.1, 0.4), gamma = c(0, 10) ), init_grid_dt = NULL, init_points = 10, # number of random points to start search n_iter = 20, # number of iterations after initial random points are set acq = 'ucb', kappa = 2.576, eps = 0.0, verbose = TRUE )  • This is a good approach, but there is a caveat: the R package rBayesianOptimization, as of the latest CRAN version 1.1.0 (which hasn't been updated in over 2 years), has no tests and a more restrictive license than the Python package by the original authors of the method, which has tests. See github.com/fmfn/BayesianOptimization. – egnha Nov 4 '18 at 15:10 This is an older question but thought I would share how I tune xgboost parameters. I originally thought I would use caret for this but recently found an issue handling all of the parameters as well as missing values. I was also considering writing an iterating loop through different combinations of parameters but wanted it to run in parallel and would require too much time. Using gridSearch from the NMOF package provided the best from both worlds (all parameters as well as parallel processing). Here is example code for binary classification (works on windows and linux): # xgboost task parameters nrounds <- 1000 folds <- 10 obj <- 'binary:logistic' eval <- 'logloss' # Parameter grid to search params <- list( eval_metric = eval, objective = obj, eta = c(0.1,0.01), max_depth = c(4,6,8,10), max_delta_step = c(0,1), subsample = 1, scale_pos_weight = 1 ) # Table to track performance from each worker node res <- data.frame() # Simple cross validated xgboost training function (returning minimum error for grid search) xgbCV <- function (params) { fit <- xgb.cv( data = data.matrix(train), label = trainLabel, param =params, missing = NA, nfold = folds, prediction = FALSE, early.stop.round = 50, maximize = FALSE, nrounds = nrounds ) rounds <- nrow(fit) metric = paste('test.',eval,'.mean',sep='') idx <- which.min(fit[,fit[[metric]]]) val <- fit[idx,][[metric]] res <<- rbind(res,c(idx,val,rounds)) colnames(res) <<- c('idx','val','rounds') return(val) } # Find minimal testing error in parallel cl <- makeCluster(round(detectCores()/2)) clusterExport(cl, c("xgb.cv",'train','trainLabel','nrounds','res','eval','folds')) sol <- gridSearch( fun = xgbCV, levels = params, method = 'snow', cl = cl, keepNames = TRUE, asList = TRUE ) # Combine all model results comb=clusterEvalQ(cl,res) results <- ldply(comb,data.frame) stopCluster(cl) # Train model given solution above params <- c(sol$minlevels,objective = obj, eval_metric = eval)
xgbModel <- xgboost(
data = xgb.DMatrix(data.matrix(train),missing=NaN, label = trainLabel),
param = params,
nrounds = results[which.min(results[,2]),1]
)

print(params)
print(results)