6
$\begingroup$

I am trying to forecast a time series using random forest. In order to validate my model, I came cross two tests: OOB and K-fold cross validation. My question is:

  1. For a time series, is out-of-bag error a out of sample test, does it make sense to test the accuracy "backward" using the historical data when I am trying to find the prediction accuracy?

  2. What is the difference between holding the last few points of the data and do a out of sample test vs a OOB?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I think on ts or any other ordered data you can only have structured cv, i.e. take the first $n-k$ observations for training, the remaining $k$ for cross-validation $\endgroup$
    – Alex
    Sep 27, 2016 at 14:52
  • $\begingroup$ @Alex Agree. I read a few documents and people are referring out-of-bag "Prediction accuracy" indicator which confused me. Maybe OOB doesn't apply to time series after all ... $\endgroup$
    – butterbetter
    Sep 28, 2016 at 14:30
  • $\begingroup$ just use cross-validation error. In sklearn package you can use GridSearchCV class for this $\endgroup$
    – Alex
    Sep 28, 2016 at 14:53

1 Answer 1

Reset to default
4
$\begingroup$

OOB error will give a misleading indication of performance on a time-series dataset because it will be evaluating performance on past data using future data. This does not give a good indication of the model's ability to perform on future data. Therefore, use a methodology like TimeSeriesSplit.

By holding out future data points for model evaluation, you examine your model's ability to perform in the future. To build your intuition on why it's "cheating" to evaluate your forecasting ability using future data points: suppose I ask you to predict my weight a year from now, given measurements I took over the past 365 days. That might be challenging, unless you see a clear pattern. Now suppose that I ask you to predict my weight on day 155 of the 365 days I already recorded, and all I do is hold out day 155. You could easily estimate my weight as (w[154] + w[156]) / 2. You just took the average of the weights between two days. But did you forecast the future? (No.) Does a measurement of your success imply you can predict my future weight? (Certainly not!)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.