I am in the process of tuning the features for my xgboost such as ordinal (label) encoding and one-hot encoding. For example, run the model with column A one-hot encoded, then run it label encoded and check the RMSE. For one of my iterations, the model improved by 5% (3,800 to 3,607 RMSE). This was strange to me because most of the iterations I tested would improve/detract the model by 1%. As in 5% was an outlier. I thought it might be due to the small sample size I had and thus there may be some overfitting.

I have a sample size of 648 observations. I split those into an 80/20 train and test set. Then I perform 5-fold cross validation on a grid-search of parameters. For those that do not want to do the math, that means 518/130, so 518 used to train and 130 observations test. Then 518 split again 80/20 is 414/104. I have also included my grid-search of parameters below and best parameters found.

What should I do to make sure that there is the least amount of overfitting? Should I increase my k for cross validation? Try to adjust gamma or max_delta_step (which I do not fully understand)? I know that is open ended, but what are the optimal parameters to tune to adjust for overfitting for small datasets?

model = XGBRegressor(booster ='gbtree', random_state = 13)
mymodel = Pipeline(steps = [('preprocessor', preprocessor),
                            ('model', model)

# Set up Parameters for CV and Grid-Search                
param_grid = dict(
                  model__gamma = [0],
                  model__n_estimators = [100,500,1000],
                  model__max_delta_step =[0],
                  model__max_depth = [5,6,7],
                  model__learning_rate= [0.1,0.3,0.5,1],
                  model__min_child_weight= [1,3,5],
                  model__early_stopping_rounds = [42],
                  model__num_parallel_tree = [1,3]
# scoring methods
sm = ['neg_mean_squared_error']
gs = GridSearchCV(mymodel
                  ,param_grid = param_grid
                  ,scoring = sm[0]
                  ,n_jobs = -1
                  ,cv = 5
                  ,refit = sm[0]
# best parameters
{'model__colsample_bytree': 0.8,
 'model__early_stopping_rounds': 42,
 'model__gamma': 0,
 'model__learning_rate': 0.1,
 'model__max_delta_step': 0,
 'model__max_depth': 5,
 'model__min_child_weight': 5,
 'model__n_estimators': 100,
 'model__num_parallel_tree': 1}

1 Answer 1


It is difficult to give a smooth answer without having the data and all required subject knowledge at hand. Still I can throw in some comments.

  1. Your validation strategy is fine as long as you decide everything by cross validation and not by the test data (but see 5.).

  2. If your best solution picks value at the border, then this is usually not a good grid. It happens for several of your parameters.

  3. For such small data, tree depth 5+ seems too much.

  4. XGBoost has important additional regularization parameters like l1 and l2 penalties. Usually these need to be tuned as well.

  5. Are the rows really independent or are there clusters of rows that invalidate your validation strategy?

  • $\begingroup$ Okay so I will reduce tree depth and introduce l1 and l2 penalties. Can you explain picking values at the border? Are you saying the best solution is picking the first value within the lists? How would I fix that other than testing more values? Also, the rows are independent. What do you mean decide everything by cv? As in use the model that provides the best training RMSE? $\endgroup$ Commented Apr 25, 2020 at 17:07
  • 2
    $\begingroup$ By picking values at the borders, I mean e.g. if your leaf size grid values are 1, 3 and 5 and 5 seems to be best, then you might extend the grid to larger values. Maybe 10 would be better. The grid should not be too small. $\endgroup$
    – Michael M
    Commented Apr 25, 2020 at 17:18
  • $\begingroup$ Increase the parameter grid, got it. Can you explain what you mean by "decide everything by cv"? As in use the model that provides the best training RMSE? $\endgroup$ Commented Apr 25, 2020 at 18:06
  • 2
    $\begingroup$ Decisions like if dummy variables are better than integer encodings should also be made by validation, not by looking at the test data set. No decisions on the model are to be made on the test data. $\endgroup$
    – Michael M
    Commented Apr 25, 2020 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.