# xgboost overfitting?

I'm trying to fit a xgboost model to some data and am getting the following results using a random 70/30 split of train/test data:

[1] train-rmse:885.071777   test-rmse:4659.329102
[2] train-rmse:763.531128   test-rmse:4633.854980
[3] train-rmse:669.896545   test-rmse:4616.207031
[4] train-rmse:599.979797   test-rmse:4603.926270
[5] train-rmse:546.986206   test-rmse:4595.299316
[6] train-rmse:508.586578   test-rmse:4589.103027
[7] train-rmse:479.262634   test-rmse:4584.626953
[8] train-rmse:457.594482   test-rmse:4581.324707
[9] train-rmse:441.262756   test-rmse:4578.702637
[10]    train-rmse:429.211090   test-rmse:4576.885742
[11]    train-rmse:419.566010   test-rmse:4575.384277
[12]    train-rmse:412.783600   test-rmse:4574.195801
[13]    train-rmse:407.107574   test-rmse:4573.338379
[14]    train-rmse:402.664185   test-rmse:4572.566895
[15]    train-rmse:399.317749   test-rmse:4572.004395


I would tend to think that this indicates over-fitting - if so what parameters could I tune to reduce the cross-validated error? No matter what parameters I change it doesn't make much difference. I have adjusted eta, min_child_weight, max_depth all with no significant impact on the cv error. Any ideas would be much appreciated.

• We'll need more details to give more specific advice (e.g, what kind of data you're dealing with, how many features, how many samples, etc.). From what you've shown, it looks like the test error is still decreasing - why not try building more than 15 trees? Jul 31, 2017 at 11:45
• How many observations are in train data set? Oct 3, 2017 at 13:24

Based on next to no info, my guesses would be:

1) Your train and test data are way too different, or you don't have enough of each.

2) You're not using the correct metric to minimise.

3) The method of train/test split is incorrect. (See one)

4) You're not applying CV correctly.

In no particular order.

I dont understand how this can be overfitting. For me overfitting occurs when you cannot generalize anymore. Here you test-rmse keeps decreasing which means that you have not overfitted yet. Increase the capacity of the model and increase the boosting rounds until you have seen test-rmse decrease and then increase. This is the behaviour you should spot

Yes, you are right, your output of xgb.cv may indicate over-fitting.

You should change parameters lambda and alpha which control L2 and L1 regularization and/or use early stopping.

This will make your model more general, less fitted to training dataset and thus more suited to predict on test dataset. Since RMSE on test set is still decreasing, you could use more iterations.