How to find the best set of parameters for XGboost I am using XGboost as a learning engine and I am getting a good results with default configurations. Now, I want to improve the predication by tuning the parameters, however, the list of parameters is pretty long. Is there a systematic way to find the best set of parameters? and which parameters could significantly affect the prediction accuracy?
 A: If you're using XGBoost within R, then you could use the caret package to fine tune the hyper-parameters.
Here is an example tuning run using caret:
library(caret)
library(xgboost)

# training set is stored in sparse matrix: devmat
myparamGrid <- expand.grid(max_depth = c(1,2,3,4,5)^2 ,
                          eta = seq(from=0.1, to=1, by=0.1),
                          nrounds=c(600,300,100) )

fitControl <- trainControl(
                           method = "repeatedcv"
                           , number = 10  # 2-fold CV
                           , repeats = 5 # repeated 5 times
                           , verboseIter=TRUE
                           , returnData=FALSE )

tuneFit <- train( devmat , factor(responseVarDev)
        , method = "xgbTree"
        , metric = "Kappa" # "Accuracy" #
        , trControl = fitControl
        , tuneGrid = myparamGrid
        )

print(tuneFit)
best_Model <- tuneFit$finalModel

# Kappa was used to select the optimal model using  the largest value.
## plot output:
    out <- tryCatch( {
        trellis.par.set(caretTheme())
        plot(tuneFit)
    }, error=myerrorHandler )
    out <- tryCatch( {
        trellis.par.set(caretTheme())
        plot(tuneFit, metric = "Kappa")
    }, error=myerrorHandler )

You should also refer to the R manual pages for caret and xgboost, they provide very good examples and pointers.
For Python, (although I haven't used this particular module) you could try the grid search classes form sklearn.grid_search , refer to this list: http://scikit-learn.org/stable/modules/classes.html#module-sklearn.grid_search .
See this link for an example of using this module for finding the best hyper-parameters for a Random Forest model - http://scikit-learn.org/stable/auto_examples/model_selection/randomized_search.html#example-model-selection-randomized-search-py 
