I'm new to machine learning and trying to train a Random Forest with time series data. I used a time series split to generate my training set and test set. In addition, because I'm working with time series data, in order to verify the robustness of the model, I am doing a walk forward test starting with 50% of the data.

So my first walk forward step is training the first 50% of the data and use the next 10% as my test set. and then for the next step, I used the first 60% of the data and use the next 10% of my test set. Repeat the steps until the end.

The oob scores are always around 63%. but the test set accuracy are all over the places(not very stable) it ranges between .48 to .63 for different steps. Is it because the RF is overfitted? Am I missing some important features ? I have 8 features and roughly 30K data points in total. I'm using 2000 trees, max features .5, min_samples_leaf 10 and max_depth 10. Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data?

Thanks guys

  • $\begingroup$ It is relatively hard to overfit with a random forest. Max depth is a little low, min samples is a little high, and max features candidated is crazy low. Could you move that up to the late 80's? What does the convergence look like that drives 2000 trees? For many cases 200 is overkill and 80 to 100 will do. Eight features is not a lot of features for 30k samples. You could augment with a hundred first lags (or leads depending on ordering) to get a better model. I don't see you talking about how 3 measurements ago informs next prediction. $\endgroup$ – EngrStudent - Reinstate Monica Jul 19 '19 at 18:09
  • $\begingroup$ @EngrStudent 2000 trees might be overkill in the sense of wasted time/computation, but adding too many trees doesn't cause overfitting under RF. $\endgroup$ – Ceph Jul 19 '19 at 20:31
  • $\begingroup$ There should be purpose in decisions. Are the classes balanced? I hear you talking about time series, but I don't see how you are handling lags. Can you talk about the shape of your data? If I had 30k rows and 8 features, I would trade it in for 29k rows and 8000 features. That is data more "fit" for a random forest. I would use Boruta to chew that 8k features to 40 features that would do some real good. $\endgroup$ – EngrStudent - Reinstate Monica Jul 25 '19 at 0:30

The OOB score is not "looking forward" to the next 10 %. For each observation it takes all the trees where it was not used for training this tree to predict this observation. So it uses also observations for training that come after this observation in a time series.

It can be used as alternative to normal cross-validation.

  • $\begingroup$ This answer doesn't seem to shed much light on why the OOB score is not serving as a more accurate estimate of the test error -- which seems to me to be OP's question. $\endgroup$ – Ceph Jul 19 '19 at 20:30
  • $\begingroup$ This answer does really involve the problem. Because it is used to test other trees, the OOB error is a "validation", which is part of the training/tuning process and it is not a "test" which is a set of data that is pristine: never once been touched by the learning process. Only "test" is going to show real-world performance; this is why the test set is retained in the untouched state. $\endgroup$ – EngrStudent - Reinstate Monica Jul 23 '19 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.