1
$\begingroup$

I'm trying to fit a xgboost model to some data and am getting the following results using a random 70/30 split of train/test data:

[1] train-rmse:885.071777   test-rmse:4659.329102 
[2] train-rmse:763.531128   test-rmse:4633.854980 
[3] train-rmse:669.896545   test-rmse:4616.207031 
[4] train-rmse:599.979797   test-rmse:4603.926270 
[5] train-rmse:546.986206   test-rmse:4595.299316 
[6] train-rmse:508.586578   test-rmse:4589.103027 
[7] train-rmse:479.262634   test-rmse:4584.626953 
[8] train-rmse:457.594482   test-rmse:4581.324707 
[9] train-rmse:441.262756   test-rmse:4578.702637 
[10]    train-rmse:429.211090   test-rmse:4576.885742 
[11]    train-rmse:419.566010   test-rmse:4575.384277 
[12]    train-rmse:412.783600   test-rmse:4574.195801 
[13]    train-rmse:407.107574   test-rmse:4573.338379 
[14]    train-rmse:402.664185   test-rmse:4572.566895 
[15]    train-rmse:399.317749   test-rmse:4572.004395 

I would tend to think that this indicates over-fitting - if so what parameters could I tune to reduce the cross-validated error? No matter what parameters I change it doesn't make much difference. I have adjusted eta, min_child_weight, max_depth all with no significant impact on the cv error. Any ideas would be much appreciated.

$\endgroup$
2
  • $\begingroup$ We'll need more details to give more specific advice (e.g, what kind of data you're dealing with, how many features, how many samples, etc.). From what you've shown, it looks like the test error is still decreasing - why not try building more than 15 trees? $\endgroup$
    – rinspy
    Jul 31, 2017 at 11:45
  • $\begingroup$ How many observations are in train data set? $\endgroup$
    – Michael M
    Oct 3, 2017 at 13:24

3 Answers 3

5
$\begingroup$

More information about your problem would be beneficial.

Based on next to no info, my guesses would be:

1) Your train and test data are way too different, or you don't have enough of each.

2) You're not using the correct metric to minimise.

3) The method of train/test split is incorrect. (See one)

4) You're not applying CV correctly.

In no particular order.

$\endgroup$
1
$\begingroup$

I dont understand how this can be overfitting. For me overfitting occurs when you cannot generalize anymore. Here you test-rmse keeps decreasing which means that you have not overfitted yet. Increase the capacity of the model and increase the boosting rounds until you have seen test-rmse decrease and then increase. This is the behaviour you should spot

$\endgroup$
0
$\begingroup$

Yes, you are right, your output of xgb.cv may indicate over-fitting.

You should change parameters lambda and alpha which control L2 and L1 regularization and/or use early stopping.

This will make your model more general, less fitted to training dataset and thus more suited to predict on test dataset. Since RMSE on test set is still decreasing, you could use more iterations.

You could read about these techniques here:

  1. high-level description of regularization in xgboost,
  2. early stopping with examples in Python,
  3. Elements of Statistical Learning - although this position does not cover xgboost implementation there is a chapter about regularization in boosted trees.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.