If I'm using a machine learning model (e.g. boosted regression trees like gbm in R) on a dataset, what does it mean if there's a significant difference between the OOB estimated optimal # of iterations and the optimal # of iterations on the test set (I'm holding out 20% of the data)?
I ask because I'm trying to model a time series response variable, and when I take the training set and sample from it without replacement, then run gbm, I get pretty different OOB and test set optimal iterations. However, when I leave the training set in its original form (oldest date to newest date), the OOB and test set optimal iterations are much closer. I'm beginning to wonder if the more recent data I have is quite a bit different from the old data.
Again general question so let me know if you need more info and I will clarify as I can.