I'm new to machine learning and trying to train a Random Forest with time series data. I used a time series split to generate my training set and test set. In addition, because I'm working with time series data, in order to verify the robustness of the model, I am doing a walk forward test starting with 50% of the data.
So my first walk forward step is training the first 50% of the data and use the next 10% as my test set. and then for the next step, I used the first 60% of the data and use the next 10% of my test set. Repeat the steps until the end.
The oob scores are always around 63%. but the test set accuracy are all over the places(not very stable) it ranges between .48 to .63 for different steps. Is it because the RF is overfitted? Am I missing some important features ? I have 8 features and roughly 30K data points in total. I'm using 2000 trees, max features .5, min_samples_leaf 10 and max_depth 10. Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data?