How to find the best set of parameters for XGboost

I am using XGboost as a learning engine and I am getting a good results with default configurations. Now, I want to improve the predication by tuning the parameters, however, the list of parameters is pretty long. Is there a systematic way to find the best set of parameters? and which parameters could significantly affect the prediction accuracy?

• I edited your title so people won't confuse your question with asking "how to use XGboost in python". – Tim Sep 21 '16 at 8:49

If you're using XGBoost within R, then you could use the caret package to fine tune the hyper-parameters.

Here is an example tuning run using caret:

library(caret)
library(xgboost)

# training set is stored in sparse matrix: devmat
myparamGrid <- expand.grid(max_depth = c(1,2,3,4,5)^2 ,
eta = seq(from=0.1, to=1, by=0.1),
nrounds=c(600,300,100) )

fitControl <- trainControl(
method = "repeatedcv"
, number = 10  # 2-fold CV
, repeats = 5 # repeated 5 times
, verboseIter=TRUE
, returnData=FALSE )

tuneFit <- train( devmat , factor(responseVarDev)
, method = "xgbTree"
, metric = "Kappa" # "Accuracy" #
, trControl = fitControl
, tuneGrid = myparamGrid
)

print(tuneFit)
best_Model <- tuneFit\$finalModel

# Kappa was used to select the optimal model using  the largest value.
## plot output:
out <- tryCatch( {
trellis.par.set(caretTheme())
plot(tuneFit)
}, error=myerrorHandler )
out <- tryCatch( {
trellis.par.set(caretTheme())
plot(tuneFit, metric = "Kappa")
}, error=myerrorHandler )


You should also refer to the R manual pages for caret and xgboost, they provide very good examples and pointers.

For Python, (although I haven't used this particular module) you could try the grid search classes form sklearn.grid_search , refer to this list: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.grid_search .

See this link for an example of using this module for finding the best hyper-parameters for a Random Forest model - http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html#example-model-selection-randomized-search-py