I am trying to build a forecasting model for the passenger vehicles registrations in a given country, and I wanted to use $auto.arima$ function from the $forecast$ package to estimate a simple ARIMA model and use it as a benchmark. However, when I analyze the forecast's accuracy, the results in the test set seem to be better than in the training set. Specifically, the MAPE is smaller in the test set than in the training set. I though that was impossible, since the ARIMA model would be perfectly fitted for the train set and its performance in the test set would be, at most, the same as in the train set.

Here is the code I used to do the analysis. The data (montlhy passenger cars registrations in Spain between 1990-2014) can be downloaded from: https://www.dropbox.com/s/jchg6gtxsgsuqqf/data.csv?dl=0

# Read the file, create a time series variable and split it in train and test sets
raw <- read.csv("data.csv")
data <- ts(raw[,2], start = 1990, frequency = 12)
training <- window(data, end = c(2013,12))
testing <- window(data, start = c(2014,1))

# Fit the model, run the forecast and measure its accuracy
fit <- auto.arima(training)
predictions <- forecast(fit, h = 12)
accuracy(predictions, testing)  

These are the results

                    ME      RMSE      MAE        MPE     MAPE      MASE        ACF1 Theil's U
Training set   81.1935 11039.568 8275.461 -0.8806572 9.982343 0.6212209 0.005405274        NA
Test set     3288.8417  6161.516 5195.148  3.4374603 7.306273 0.3899885 0.077836661 0.3779246

I don't know if that helps, but the same happens if I use a log transformation on the data and also when using setting stepwise = FALSE and approximation = FALSE, which results in a more accurate ARIMA model.

Is it a coding error? If it's not an error, how can this result be interpreted?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.