What would be the fair way of comparing ARIMA vs LSTM forecast?

Question

For a time series forecasting program, I'm experimenting with LSTMs and trying to compare the error obtained with the error corresponding to traditional forecasting methods, like Holt and ARIMA.

If I divide my set into training and testing (30 days) for the LSTM, what would be the correct way to forecast with Holt and ARIMA (for the same testing period), so that the RMSE comparison is fair:

test = dataset[-test_size:] 
test['holt'] = np.nan
for i in range(0,test_size):    
    training = dataset[0:-test_size+i+1]
    holt = Holt(np.asarray(training)).fit(smoothing_level = 0.9,smoothing_slope = 0.1)
    test['holt'][i] = holt.forecast(1)

Or,

test = dataset[-test_size:] 
test['holt'] = np.nan
training = dataset[0:-test_size]
holt = Holt(np.asarray(training)).fit(smoothing_level = 0.9,smoothing_slope = 0.1)
test['holt'] = holt.forecast(test_size)

In other words, should the traditional method readjust (and "retrain") for each day that it predicts, taking the real value into account for the next prediction, or should I forecast the 30 days at once, only using the "training" set?

Answer 1

Either approach makes sense. One is commonly called "fixed origin" forecasting, the other "rolling origin", or (less commonly) "time series cross-validation". Just make sure to clearly state what you are doing, since the two approaches cannot be compared.

The section on evaluating forecast accuracy from FPP2 may be helpful.