1
$\begingroup$

I'm new to machine learning and trying to train a Random Forest with time series data. I used a time series split to generate my training set and test set. In addition, because I'm working with time series data, in order to verify the robustness of the model, I am doing a walk forward test starting with 50% of the data.

So my first walk forward step is training the first 50% of the data and use the next 10% as my test set. and then for the next step, I used the first 60% of the data and use the next 10% of my test set. Repeat the steps until the end.

The oob scores are always around 63%. but the test set accuracy are all over the places(not very stable) it ranges between .48 to .63 for different steps. Is it because the RF is overfitted? Am I missing some important features ? I have 8 features and roughly 30K data points in total. I'm using 2000 trees, max features .5, min_samples_leaf 10 and max_depth 10. Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data?

Thanks guys

Improve this question
$\endgroup$
3
  • $\begingroup$ It is relatively hard to overfit with a random forest. Max depth is a little low, min samples is a little high, and max features candidated is crazy low. Could you move that up to the late 80's? What does the convergence look like that drives 2000 trees? For many cases 200 is overkill and 80 to 100 will do. Eight features is not a lot of features for 30k samples. You could augment with a hundred first lags (or leads depending on ordering) to get a better model. I don't see you talking about how 3 measurements ago informs next prediction. $\endgroup$
    – EngrStudent
    Commented Jul 19, 2019 at 18:09
  • $\begingroup$ @EngrStudent 2000 trees might be overkill in the sense of wasted time/computation, but adding too many trees doesn't cause overfitting under RF. $\endgroup$
    – Ceph
    Commented Jul 19, 2019 at 20:31
  • $\begingroup$ There should be purpose in decisions. Are the classes balanced? I hear you talking about time series, but I don't see how you are handling lags. Can you talk about the shape of your data? If I had 30k rows and 8 features, I would trade it in for 29k rows and 8000 features. That is data more "fit" for a random forest. I would use Boruta to chew that 8k features to 40 features that would do some real good. $\endgroup$
    – EngrStudent
    Commented Jul 25, 2019 at 0:30

2 Answers 2

Reset to default
1
$\begingroup$

The OOB score is not "looking forward" to the next 10 %. For each observation it takes all the trees where it was not used for training this tree to predict this observation. So it uses also observations for training that come after this observation in a time series.

It can be used as alternative to normal cross-validation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This answer doesn't seem to shed much light on why the OOB score is not serving as a more accurate estimate of the test error -- which seems to me to be OP's question. $\endgroup$
    – Ceph
    Commented Jul 19, 2019 at 20:30
  • $\begingroup$ This answer does really involve the problem. Because it is used to test other trees, the OOB error is a "validation", which is part of the training/tuning process and it is not a "test" which is a set of data that is pristine: never once been touched by the learning process. Only "test" is going to show real-world performance; this is why the test set is retained in the untouched state. $\endgroup$
    – EngrStudent
    Commented Jul 23, 2019 at 14:49
1
$\begingroup$

Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?

Goal:
The solution from training MUST "work well" on the validation set from a different time series.

There are two Issues:

  1. We could just be overfitting
  2. We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)
  3. Bit of BOTH

Thesis:

The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).

Having said that, it is still a useful metric to check out which of the two issues you are having. This allows us to see whether the model is overfitting, without needing a separate validation set.

Interpretation Example

In this lecture they show a training error of 17%, OOB error of 21% and a validation error of 23%.

The training error is LESS THAN OOB error indicating overfitting. Because the OOB error is "much less" than the validation error, we can gather that there is "something else" in addition to overfitting (i.e., perhaps something related to time).

Conclusion:

Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.

training error < OOB --> overfitting
OOB != validation --> domain shift

P.S.

  1. Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?
  2. I borrowed this heavily from Jeremy Howard (founder of fastai)from his course on Deeplearning.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.