2
$\begingroup$

I am doing grid based parameter search using CV for my XGBoost model in R. Code is for reference but question below is lot generic:

opt1 <- as.integer(3/(sum(base_data$NON_CLR_SLS_Q > 0)/nrow(base_data)))
opt2 <- as.integer(1/sqrt(sum(base_data$NON_CLR_SLS_Q > 0)/nrow(base_data)))
searchGridSubCol <- as.data.table(expand.grid(subsample = c(0.75, 1),
                             colsample_bytree = c(0.6, 0.8),
                             min_child_weight = c(opt2)
                             ))

cross_val <- function(parameterList){

 #Extract Parameters to test
 currentSubsampleRate <- parameterList$subsample
 currentColsampleRate <- parameterList$colsample_bytree
 current_min_child_weight <- parameterList$min_child_weight

param1 <- list("objective" = "reg:linear",
               "eval_metric" = "rmse",
               "max_depth" = 10,
               "eta" = 0.03,
               "gamma" = 0,
               "subsample" = currentSubsampleRate,
              "colsample_bytree" = currentColsampleRate,
               "min_child_weight" = current_min_child_weight,
               "alpha" = 0.1)

 xgboostModelCV <- xgb.cv(data=train_matrix, label=output_train, nrounds = 500, nfold = 5,
                         verbose = F, early.stop.round=5, prediction=T, param=param1, missing = NaN)

 test123 <- data.table(rowSubSample=currentSubsampleRate,colSubSample=currentColsampleRate,
   min_child_weight=current_min_child_weight,
   numIter=which.min(xgboostModelCV$dt[, test.rmse.mean]),
   RMSE=xgboostModelCV$dt[which.min(xgboostModelCV$dt[, test.rmse.mean]),test.rmse.mean])

 rm(xgboostModelCV)

 return(test123)

 }

rmseErrorsHyperparameters <- searchGridSubCol[,cross_val(.SD),by=1:nrow(searchGridSubCol)]

rmseErrorsHyperparameters <- rmseErrorsHyperparameters[order(rmseErrorsHyperparameters$RMSE),]

rmseErrorsHyperparameters <- head(rmseErrorsHyperparameters,1)

I ran my model for nrounds = 500 but time taken when using grid combinations in (unexpectedly) non-linearly growing. So I want to cut time for grid search.

I found 500 is good limit for grid search in my case, as most cases model overfitting start before 500 iterations. So this is full blown CV. However to save time I want to just run each combination for say 100 or 200 iterations and compare (this I am calling limited CV). What is shortcoming of limited CV Vs full blown CV for grid search; apart from not be able to get statistically sound estimate for RMSE (in my case). Can optimal parameters of limited CV can be different than full blown CV if yes then why? In other words is it always necessary to run till overfitting is detected when doing hyperparameter tuning?

$\endgroup$
1
$\begingroup$

This answer is based on a general knowledge of hyperparameters (most specific with experience on SVM hyperparameters) and not any specific knowledge on gradient boosting or the xgboost implementation in particular.

Hyperparameters are not in general independent, that is one can fix all but one of them, search for the best choice for the remaining hyperparameters, and repeat the process. The OP is not doing this, but he/she is assuming that limiting the range of one number of interactions, and searching for the others will result in the same solution one would get without the limit on the interactions. This is not likely to be true given my experience.

If search time is an issue I would reduce the number of folds in the grid search. The larger the dataset, lower is the variance on the CV estimation, so you could probably get the same or very similar hyperparameters with 3-fold, and I would go as low as 2-fold. I have some experiments that 2-fold is OK for SVM, with datasets above 1k.

$\endgroup$
1
  • $\begingroup$ I know that hyperparameters are dependent on each other however in order to cut short time of all parameter grid tuning I have already done some experimentation and found hard coded parameters mostly good. Unlike SVM where I guess one tune 3-4 parameters XGBoost have 7-8 parameters to tune which should be dealt prudently. Upvote for lesser fold idea. $\endgroup$ – abhiieor Oct 12 '16 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.