I am using XGboost as a learning engine and I am getting a good results with default configurations. Now, I want to improve the predication by tuning the parameters, however, the list of parameters is pretty long. Is there a systematic way to find the best set of parameters? and which parameters could significantly affect the prediction accuracy?

  • 2
    $\begingroup$ I edited your title so people won't confuse your question with asking "how to use XGboost in python". $\endgroup$ – Tim Sep 21 '16 at 8:49

If you're using XGBoost within R, then you could use the caret package to fine tune the hyper-parameters.

Here is an example tuning run using caret:


# training set is stored in sparse matrix: devmat
myparamGrid <- expand.grid(max_depth = c(1,2,3,4,5)^2 ,
                          eta = seq(from=0.1, to=1, by=0.1),
                          nrounds=c(600,300,100) )

fitControl <- trainControl(
                           method = "repeatedcv"
                           , number = 10  # 2-fold CV
                           , repeats = 5 # repeated 5 times
                           , verboseIter=TRUE
                           , returnData=FALSE )

tuneFit <- train( devmat , factor(responseVarDev)
        , method = "xgbTree"
        , metric = "Kappa" # "Accuracy" #
        , trControl = fitControl
        , tuneGrid = myparamGrid

best_Model <- tuneFit$finalModel

# Kappa was used to select the optimal model using  the largest value.
## plot output:
    out <- tryCatch( {
    }, error=myerrorHandler )
    out <- tryCatch( {
        plot(tuneFit, metric = "Kappa")
    }, error=myerrorHandler )

You should also refer to the R manual pages for caret and xgboost, they provide very good examples and pointers.

For Python, (although I haven't used this particular module) you could try the grid search classes form sklearn.grid_search , refer to this list: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.grid_search .

See this link for an example of using this module for finding the best hyper-parameters for a Random Forest model - http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html#example-model-selection-randomized-search-py


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.