10
$\begingroup$

I'm attempting to utilise RF regression to make predictions on the performance of a paper mill.

I have minute by minute data for the inputs (rate and amount of wood pulp going in etc...) as well as for the performance of the machine (paper produced, power drawn by the machine) and am looking to make predictions 10 minutes ahead on the performance variables.

I've got 12 months of data, so have separated it into 11 months for the training set, and the final month for testing.

So far I have created 10 new features which are lagged values by 1-10 minutes for each of the performance variables, and used these as well as the inputs to make predictions. The performance on the test set has been quite good ( the system is quite predictable), but I'm worried that I'm missing something in my approach.

For example, in this paper, the authors state their approach in testing the predictive ability of their random forest model:

The simulation proceeds by iteratively adding a new week of data, training a new model based on the updated data, and predicting the number of outbreaks for the following week

How is this different from utilizing 'later' data in the time series as testing? Should I be validating my RF regression model with this approach as well as on the testing data set? Furthermore, is this sort of 'autoregressive' approach to random forest regression valid for time series, and do I even need to create this many lagged variables if I'm interested in a prediction 10 minutes in the future?

$\endgroup$
  • 2
    $\begingroup$ RFs are not designed for and do not explicitly integrate temporal considerations. Given that, why use them at all for this analysis? There are many time series methodologies out there. Pick one. $\endgroup$ – Mike Hunter Feb 22 '17 at 13:15
  • 2
    $\begingroup$ @DJohnson I thought I would attempt to mimic the approach in the paper: attempt RF and compare it to ARIMA. Are you suggesting this is not worth the time and to just utilise ARIMA? $\endgroup$ – KRS-fun Feb 22 '17 at 13:21
  • 4
    $\begingroup$ @DJohnson, the mechanics of autoregressive models are much like that of cross-sectional regression models. Once the lagged features are constructed, why not use RFs as in a cross-sectional setting? I think it is fair to try them out. But you are right that other methods are more popular in time series, and the OP might benefit from exploring them, too. $\endgroup$ – Richard Hardy Feb 22 '17 at 13:24
  • 1
    $\begingroup$ My view is that RFs are like the hammer where everything becomes a nail. With data as described by the OP, my first choice would be a panel data or pooled model, not ARIMA. $\endgroup$ – Mike Hunter Feb 22 '17 at 13:55
  • 5
    $\begingroup$ I came across this just now, and read the paper referred to a couple days ago. I am comparing random forest and an LSTM for multivariate time series forecasting. Interestingly, the LSTM does better when including less time in the training data, but as I add in more years of data, the results of both methods are converging to the true results. I think this is mostly because the features provide enough information to overcome the temporal component. Anyways, thought it was interesting. Also, I've never seen ARIMA work well except in very obvious seasonal cases, and multivariate ARIMA is... $\endgroup$ – Hobbes Jun 14 '17 at 21:55
6
$\begingroup$

How is this different from utilizing 'later' data in the time series as testing?

The approach you quote is called "rolling origin" forecasting: the origin from which we forecast out is "rolled forward", and the training data is updated with the newly available information. The simpler approach is "single origin forecasting", where we pick a single origin.

The advantage of rolling origin forecasting is that it simulates a forecasting system over time. In single origin forecasting, we might by chance pick an origin where our system works very well (or very badly), which might give us an incorrect idea of our system's performance.

One disadvantage of rolling origin forecasting is its higher data requirement. If we want to forecast out 10 steps with at least 50 historical observations, then we can do this single-origin with 60 data points overall. But if we want to do 10 overlapping rolling origins, then we need 70 data points.

The other disadvantage is of course its higher complexity.

Needless to say, you should not use "later" data in rolling origin forecasting, either, but only use data prior to the origin you are using in each iteration.

Should I be validating my RF regression model with this approach as well as on the testing data set?

If you have enough data, a rolling origin evaluation will always inspire more confidence in me than a single origin evaluation, because it will hopefully average out the impact of the origin.

Furthermore, is this sort of 'autoregressive' approach to random forest regression valid for time series, and do I even need to create this many lagged variables if I'm interested in a prediction 10 minutes in the future?

Yes, rolling vs. single origin forecasting is valid for any predictive exercise. It doesn't depend on whether you use random forests or ARIMA or anything else.

Whether you need your lagged variables is something we can't counsel you on. It might be best to talk to a subject matter expert, who might also suggest other inputs. Just try your RF with the lagged inputs vs. without. And also compare to standard benchmarks like ARIMA or ETS or even simpler methods, which can be surprisingly hard to beat.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.