Two ways of time series cross-validation for ARIMA giving different results I have data x that I have split into train and test data:
train = x[1:80,1]
test = x[81:length(x), 1]

I am trying to do cross-validation: i.e. fit the model to train and then see how it performs compared to the test data.
The first method I'm using is through the Arima() function:
fittrain = Arima(train, order=c(1,1,0))
fittest = Arima(test, model=fittrain)
accuracy(fittest)

> accuracy(fittest)
                   ME     RMSE      MAE       MPE      MAPE      MASE
Training set 346.2611 414.9891 353.2179 0.1926218 0.1965906 0.2973771
                   ACF1
Training set -0.1857537

I wanted to confirm that this is indeed comparing the fitted model to my test data:
fit = Arima(train, order=c(1,1,0))
preds = as.vector(forecast(fit, h = length(test))$mean)
RMSE = sqrt(mean((preds - as.vector(testdata)) ^ 2))
RMSE

> RMSE
[1] 4022.871

Why am I getting such different values for the RMSE? Is there a function that will compare my model forecasts to the actual test data values and give me a bunch of different metrics?
 A: In case 1, you are looking at in-sample fit in the test sample from a model whose coefficients were optimized for the training sample rather than the test sample. I.e. you look at properties of residuals $\Delta y_t-\widehat{\Delta y}_t$ for $t$ running in the test sample where $\widehat{\Delta y}_t=\hat\varphi_1 \Delta y_{t-1}$ where $\hat\varphi_1$ is the fitted coefficient based on the training sample.
In case 2, you at looking at forecast errors from forecasts of an increasingly longer horizon. I.e. $y_{T+h}-\widehat{\Delta y}_{T+h}$ where $\widehat{\Delta y}_{T+h}=(\hat\varphi_1)^h y_T$ where $y_T$ is the last point in the training sample and $T+h$ runs in the test sample.
Clearly, you expect a much better in-sample fit in case 1 even though the coefficients were optimized on a different sample, as long as the data generating process has not changed from the training to the test sample, and a much worse forecast accuracy in case 2 since the forecasts are of an increasingly longer horizon. This is exactly what you have observed.
