Test MAPE < Train MAPE using auto.arima()

Question

I am trying to build a forecasting model for the passenger vehicles registrations in a given country, and I wanted to use $auto.arima$ function from the $forecast$ package to estimate a simple ARIMA model and use it as a benchmark. However, when I analyze the forecast's accuracy, the results in the test set seem to be better than in the training set. Specifically, the MAPE is smaller in the test set than in the training set. I though that was impossible, since the ARIMA model would be perfectly fitted for the train set and its performance in the test set would be, at most, the same as in the train set.

Here is the code I used to do the analysis. The data (montlhy passenger cars registrations in Spain between 1990-2014) can be downloaded from: https://www.dropbox.com/s/jchg6gtxsgsuqqf/data.csv?dl=0

# Read the file, create a time series variable and split it in train and test sets
raw <- read.csv("data.csv")
data <- ts(raw[,2], start = 1990, frequency = 12)
training <- window(data, end = c(2013,12))
testing <- window(data, start = c(2014,1))

# Fit the model, run the forecast and measure its accuracy
fit <- auto.arima(training)
predictions <- forecast(fit, h = 12)
accuracy(predictions, testing)  

These are the results

                    ME      RMSE      MAE        MPE     MAPE      MASE        ACF1 Theil's U
Training set   81.1935 11039.568 8275.461 -0.8806572 9.982343 0.6212209 0.005405274        NA
Test set     3288.8417  6161.516 5195.148  3.4374603 7.306273 0.3899885 0.077836661 0.3779246

I don't know if that helps, but the same happens if I use a log transformation on the data and also when using setting stepwise = FALSE and approximation = FALSE, which results in a more accurate ARIMA model.

Is it a coding error? If it's not an error, how can this result be interpreted?

Answer 1

It's hard to give a good answer, since the data is not available any more, and we don't even know what model auto.arima fitted to your data. However, here is one possible explanation.

The MAPE is not minimized in expectation by the conditional mean (which ARIMA aims for), but by a rather exotic functional of the unknown future density, the (-1)-median, which is best described tautologically as "the functional that minimizes the expected MAPE": What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?. Crucially, that functional is usually lower than the conditional mean.

Thus, it may well be that there is a trend in your data that auto.arima does not catch, or underestimates. If so, it may well be that in-sample, it gets close to the conditional mean, but out-of-sample is biased low - but therefore is closer to the (-1)-median.

The MAPE can be badly misleading. Its apparent easier interpretability makes it even more dangerous.