I am trying to build a forecasting model for the passenger vehicles registrations in a given country, and I wanted to use $auto.arima$ function from the $forecast$ package to estimate a simple ARIMA model and use it as a benchmark. However, when I analyze the forecast's accuracy, the results in the test set seem to be better than in the training set. Specifically, the MAPE is smaller in the test set than in the training set. I though that was impossible, since the ARIMA model would be perfectly fitted for the train set and its performance in the test set would be, at most, the same as in the train set.
Here is the code I used to do the analysis. The data (montlhy passenger cars registrations in Spain between 1990-2014) can be downloaded from: https://www.dropbox.com/s/jchg6gtxsgsuqqf/data.csv?dl=0
# Read the file, create a time series variable and split it in train and test sets raw <- read.csv("data.csv") data <- ts(raw[,2], start = 1990, frequency = 12) training <- window(data, end = c(2013,12)) testing <- window(data, start = c(2014,1)) # Fit the model, run the forecast and measure its accuracy fit <- auto.arima(training) predictions <- forecast(fit, h = 12) accuracy(predictions, testing)
These are the results
ME RMSE MAE MPE MAPE MASE ACF1 Theil's U Training set 81.1935 11039.568 8275.461 -0.8806572 9.982343 0.6212209 0.005405274 NA Test set 3288.8417 6161.516 5195.148 3.4374603 7.306273 0.3899885 0.077836661 0.3779246
I don't know if that helps, but the same happens if I use a log transformation on the data and also when using setting stepwise = FALSE and approximation = FALSE, which results in a more accurate ARIMA model.
Is it a coding error? If it's not an error, how can this result be interpreted?