2
$\begingroup$

For a time series forecasting program, I'm experimenting with LSTMs and trying to compare the error obtained with the error corresponding to traditional forecasting methods, like Holt and ARIMA.

If I divide my set into training and testing (30 days) for the LSTM, what would be the correct way to forecast with Holt and ARIMA (for the same testing period), so that the RMSE comparison is fair:

test = dataset[-test_size:] 
test['holt'] = np.nan
for i in range(0,test_size):    
    training = dataset[0:-test_size+i+1]
    holt = Holt(np.asarray(training)).fit(smoothing_level = 0.9,smoothing_slope = 0.1)
    test['holt'][i] = holt.forecast(1)

Or,

test = dataset[-test_size:] 
test['holt'] = np.nan
training = dataset[0:-test_size]
holt = Holt(np.asarray(training)).fit(smoothing_level = 0.9,smoothing_slope = 0.1)
test['holt'] = holt.forecast(test_size)

In other words, should the traditional method readjust (and "retrain") for each day that it predicts, taking the real value into account for the next prediction, or should I forecast the 30 days at once, only using the "training" set?

$\endgroup$
2
$\begingroup$

Either approach makes sense. One is commonly called "fixed origin" forecasting, the other "rolling origin", or (less commonly) "time series cross-validation". Just make sure to clearly state what you are doing, since the two approaches cannot be compared.

The section on evaluating forecast accuracy from FPP2 may be helpful.

$\endgroup$
  • $\begingroup$ (+1) I have found the function createTimeSlices from caret really convenient to do this kind of validation to a time-series forecasts. $\endgroup$ – usεr11852 Jul 16 '18 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.