OOB Score vs test set accuray Random Forest I'm new to machine learning and trying to train a Random Forest with time series data. I used a time series split to generate my training set and test set. In addition, because I'm working with time series data, in order to verify the robustness of the model, I am doing a walk forward test starting with 50% of the data.
So my first walk forward step is training the first 50% of the data and use the next 10% as my test set. and then for the next step, I used the first 60% of the data and use the next 10% of my test set. Repeat the steps until the end. 
The oob scores are always around 63%. but the test set accuracy are all over the places(not very stable) it ranges between .48 to .63 for different steps. Is it because the RF is overfitted? Am I missing some important features ?  I have 8 features and roughly 30K data points in total. I'm using 2000 trees, max features .5, min_samples_leaf 10 and max_depth 10. 
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? 
Thanks guys
 A: The OOB score is not "looking forward" to the next 10 %. For each observation it takes all the trees where it was not used for training this tree to predict this observation. So it uses also observations for training that come after this observation in a time series. 
It can be used as alternative to normal cross-validation. 
A: Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?
Goal:
The solution from training MUST "work well" on the validation set from a different time series.
There are two Issues:

*

*We could just be overfitting

*We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)

*Bit of BOTH

Thesis:
The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).
Having said that, it is still a useful metric to check out which of the two issues you are having. This allows us to see whether the model is overfitting, without needing a separate validation set.
Interpretation Example
In this lecture they show a training error of 17%, OOB error of 21% and a validation error of 23%.
The training error is LESS THAN OOB error indicating overfitting.
Because the OOB error is "much less" than the validation error, we can gather that there is "something else" in addition to overfitting (i.e., perhaps something related to time).
Conclusion:
Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.
training error < OOB --> overfitting
OOB != validation --> domain shift
P.S.

*

*Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?

*I borrowed this heavily from Jeremy Howard (founder of fastai)from his course  on Deeplearning.

