1
$\begingroup$

I am using the xgboost package in python to do predictive modelling problems. I encounter the following two questions:

First, I've specified a set of values for those hyper parameters and run cross validation

params1 = {
'max_depth':5,
'min_child_weight':1,
'learning_rate':0.1,
'gamma': 0,
'subsample':0.8,
'colsample_bytree':0.8,
'objective': 'binary:logistic',
'silent': 1,
'scale_pos_weight':1,
'nthread': 4,
}
num_rounds = 1000
hist = xgb.cv(params1, Dtrain, num_rounds, nfold=10, metrics={'error'},early_stopping_rounds = 50, seed=seed)
hist

enter image description here

My question is: does the xgb.cv do the following thing: for each value of the num_rounds (starting from 0 or 1?), it actually runs a 10-fold cv and outputs both the mean error of test and train? And here as we have 10 numbers for both train and test error, we could obtain the standard deviation of them. We know that the boost algorithm works as follows: it fits a simple tree in round 0 (or 1?), then in the next round, it fits another simple tree to the residual generating from the previous round, so on and so forth. Thus, it looks to me that this xgb.cv not only does cross validation, it also performs a hyper parameter tunning process for the num_round variable? Because, if it is just for cross validation, then it should just run the cv for the model with num_round=1000 as I specified, but here it seems that it does cv for each value of num_round=0,1,2,3,..... And based on the early stopping rule, it finds the "optimal" value of num_round, in this example, it is 8, given all the other hyper parameters fixed.

Then, I found that sklearn package as a GridSearchCV to conduct the hyper parameter tunning process. Then I input exactly the same value of the above hyper parameters into GridSearchCV, I want to verify that these two methods agree with each other and they should. However, the result does NOT support this conjecture

params_fixed = {
'silent': 1,
'scale_pos_weight':1,
'nthread': 4,
'objective': 'binary:logistic'
}
params_grid = {
'learning_rate': [0.1],
'n_estimators':[8],
'max_depth':[5],
'min_child_weight':[1],
'gamma':[0],
'subsample':[0.8], 
'colsample_bytree':[0.8] 
}
bst_grid = GridSearchCV(
estimator=XGBClassifier(**params_fixed, seed=seed),
param_grid=params_grid,
cv=10,
scoring='accuracy'
)
bst_grid.grid_scores_

enter image description here

Note that it first correctly recognize that there is only one set of value of those hyper parameters to search. It reports mean:0.82604. I believe, for the set of hyper parameters I've specified, this GridSearchCV runs a 10 fold cv, which gives him the average accuracy rate 0.82604 **but this number does not agree with any one of those numbers above in the table that are subtracted by 1. ** As I fixed the seed. I could not come up with any other source of randomness.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.