3
$\begingroup$

If I'm using a machine learning model (e.g. boosted regression trees like gbm in R) on a dataset, what does it mean if there's a significant difference between the OOB estimated optimal # of iterations and the optimal # of iterations on the test set (I'm holding out 20% of the data)?

I ask because I'm trying to model a time series response variable, and when I take the training set and sample from it without replacement, then run gbm, I get pretty different OOB and test set optimal iterations. However, when I leave the training set in its original form (oldest date to newest date), the OOB and test set optimal iterations are much closer. I'm beginning to wonder if the more recent data I have is quite a bit different from the old data.

Again general question so let me know if you need more info and I will clarify as I can.

$\endgroup$

1 Answer 1

2
$\begingroup$

First of all, it is normal that there is no truly optimal hyperparameter value set -- you would need a full knowledge about the process to have it, which is obviously impossible.

However if you say the differences are large, it is also possible that the train set is not representative to the sample, most likely because of some more or less systematic drifts or some rare change in the process near the split between train and test.
How to test this? Drifts are usually visible on decision(time) plots, moreover you can add time as a predictor and check if it is used by the model (although this may end in a total overfitting).
The next option is to check the results on test selected from some other period or made from randomly chosen time moments (don't do the latter if you use history). If the results will be similar, it means the model is overfitting; if not, it would suggest a change in the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.