Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?

Goal:
The solution from training MUST "work well" on the validation set from a different time series.

There are two Issues:

1. We could just be overfitting
2. We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)
3. Bit of BOTH

Thesis:

The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).

Having said that, it is still a useful metric to check out which of the two issues you are having. Is it because of out of domain data or is it because you are overfitting? This allows us to see whether the model is overfitting, without needing a separate validation set.

Interpretation Example

In this lecture they show a training error of 17%, OOB error of 21% and a validation error of 23%.

The training error is different from OOB error indicating overfitting. Because the OOB error is "much less" than the validation error, we can gather that there is "something else" in addition to overfitting (i.e., perhaps something related to time).

Conclusion:

Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.

P.S.

1. Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?
2. I borrowed this heavily from Jeremy Howard (founder of fastai)from his course on Deeplearning.

Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?

Goal:
The solution from training MUST "work well" on the validation set from a different time series.

There are two Issues:

1. We could just be overfitting
2. We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)
3. Bit of BOTH

Thesis:

The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).

Having said that, it is still a useful metric to check out which of the two issues you are having. This allows us to see whether the model is overfitting, without needing a separate validation set.

Interpretation Example

In this lecture they show a training error of 17%, OOB error of 21% and a validation error of 23%.

The training error is LESS THAN OOB error indicating overfitting. Because the OOB error is "much less" than the validation error, we can gather that there is "something else" in addition to overfitting (i.e., perhaps something related to time).

Conclusion:

Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.

training error < OOB --> overfitting
OOB != validation --> domain shift

P.S.

1. Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?
2. I borrowed this heavily from Jeremy Howard (founder of fastai)from his course on Deeplearning.

Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?

Goal:
The solution from training MUST "work well" on the validation set from a different time series.

There are two Issues:

1. We could just be overfitting
2. We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)
3. Bit of BOTH

Thesis:

The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).

Having said that, it is still a useful metric to check out which of the two issues you are having. Is it because of out of domain data or is it because you are overfitting? This allows us to see whether the model is overfitting, without needing a separate validation set.

Conclusion:

Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.

P.S.

1. Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?
2. I borrowed this heavily from Jeremy Howard (founder of fastai)from his course on Deeplearning.

Question
Is it true that oob score might not be the optimal metric to reflect the general accuracy with time series data? Is (my error) because the RF is overfitted?

Goal:
The solution from training MUST "work well" on the validation set from a different time series.

There are two Issues:

1. We could just be overfitting
2. We could be very bad at predicting "out of domain data" (e.g., our validation set from a different time series)
3. Bit of BOTH

Thesis:

The OOB accuracy captures only one of the issues (overfitting). The OOB prediction from each row of data could be in the same time series as in the training (atleast there is a high chance for every row).

Having said that, it is still a useful metric to check out which of the two issues you are having. Is it because of out of domain data or is it because you are overfitting? This allows us to see whether the model is overfitting, without needing a separate validation set.

Interpretation Example

In this lecture they show a training error of 17%, OOB error of 21% and a validation error of 23%.

The training error is different from OOB error indicating overfitting. Because the OOB error is "much less" than the validation error, we can gather that there is "something else" in addition to overfitting (i.e., perhaps something related to time).

Conclusion:

Yes it doesn't seem optimal as it is "similar" to cross validation where the time series issue would not be captured. However it is a clever way of determining if you are overfitting.

P.S.

1. Regarding your question on why the test accuracy is all over the place, perhaps can you provide the data? Maybe it is a kaggle competition?
2. I borrowed this heavily from Jeremy Howard (founder of fastai)from his course on Deeplearning.