RMSE or MAPE? which one to choose for accuracy?

I have a weekly times series for which I would like to find the best fit model. So far I've tried arima, Harmonic regression with arima error, neural network and in the end I would like to decide which one has been better fitted to my raw data. The time series look like this, with heavy seasonal and cyclic pattern: I also put the Ljung-Box test and the plot of predicted values here for each:

# Arima
fit <- training %>% auto.arima(lambda = 0)
fit %>%
checkresiduals()

Ljung-Box test

data:  Residuals from ARIMA(3,1,0)
Q* = 23.619, df = 23, p-value = 0.4252

Model df: 3.   Total lags used: 26


#Harmonic regression with arima error
fit2 <- auto.arima(training, lambda = 0, seasonal = TRUE, xreg = fourier(training, K = 4))
fit2 %>%
checkresiduals()

Ljung-Box test

data:  Residuals from Regression with ARIMA(2,1,1) errors
Q* = 21.642, df = 15, p-value = 0.1175

Model df: 11.   Total lags used: 26


#Neural Network
fit3 <- nnetar(training, lambda = 0)
fit3


They all seem fine based on Ljung-Box test but they somehow failed to capture the wiggly form of the time-series here which I don't know how important it is. But my main question is when I check the accuracy if I choose RMSE I have to pick the harmonic regression one and if I choose MAPE I have to pick neural network model. And I would also like to know why RMSE and MAPE values are so different here.

# Arima
accuracy(forecast(fit, h = 16), test)
ME     RMSE      MAE       MPE     MAPE      MASE       ACF1 Theil's U
Training set  1.948693 27.56683 19.09467 -4.402578 25.87164 0.5790763 0.21495069        NA
Test set     43.293579 61.02374 46.31065 32.745528 39.26652 1.4044442 0.09636865  1.158678

# Harmonic Regression
accuracy(forecast(h = 16, fit2, xreg = fourier(training, K = 4)), test)
ME    RMSE      MAE         MPE      MAPE      MASE      ACF1 Theil's U
Training set  4.323546 24.4800 16.05035   -1.464388  21.89874 0.4867525 0.1751586        NA
Test set     -2.495049 42.1323 33.03114 -171.095704 194.16485 1.0017220 0.2349017  4.288442

# Neural Network
accuracy(forecast(fit3, h = 16), test)

ME     RMSE      MAE       MPE     MAPE      MASE      ACF1 Theil's U
Training set  3.414448 22.63083 14.31504 -2.615375 16.93870 0.4341265 0.2253450        NA
Test set     40.095160 58.90628 44.16645 28.908539 37.72779 1.3394181 0.1107563  1.119875


Thank you very much for your help, I really appreciate it in advance.

• Phrases like "best fit" and "accuracy" are not well-defined until you say what you want them to mean. Once you do that properly, your loss function should become clear enough to operationalize. That won't solve the lack of fit problem you perceive with the models you have there, of course, because that's got little to do with the specific loss. Dec 1 '21 at 23:07
• Thank you very much for your comment. Since I am quite naive in this field, can you please elaborate how I could improve my question? or perhaps if you have any recommendations on improving models I would appreciate it. Dec 1 '21 at 23:43
• Be aware that it is hard to forecast the stock market. You will probably not be able to find a simple model that forecasts the VIX with any precision. Dec 2 '21 at 13:32
• And if you do figure out how to predict the VIX, please tell me (and only me) how to do so :p
– Dave
Dec 2 '21 at 13:48
• @Dave I will have a look and will be in touch hahaha Dec 2 '21 at 15:55

Use the RMSE.

Note that the (R)MSE and the MAPE will be minimized by quite different point forecasts (see my answer at Higher RMSE lower MAPE). You should first decide which functional of the unknown future distribution you want to elicit, then choose the corresponding error measure.

However, note that an ARIMA model will output a conditional expectation forecast, i.e., the functional that optimizes (R)MSE. It makes little sense to train a model to minimize the (R)MSE, then to assess its forecasts with a different error measure (Kolassa, 2020, IJF). If you truly want to find a MAPE-optimal forecast, you should also use the MAPE to fit your model. I am not aware of any off-the-shelf forecasting software that does this (if you use an ML pipeline, you may be able to specify any fitting criterion and choose the MAPE), and I have major doubts as to the usefulness of a MAPE-minimal forecast.

• Thank you very much Mr. Kolassa for your answer. I already visited your other answered today but thought I could ask to be sure that it also the case with my time-series. It's a general rule though. Dec 1 '21 at 20:34
• Ah, thank you for the upvote there then, I presume. You might find my IJF paper useful, it's quite short and accessible. Feel free to request it from me on ResearchGate, if you don't have access to the IJF. Dec 1 '21 at 20:36
• Great! Can you please add a few words in your LinkedIn invitation and refer to this thread? I get a lot of invitations from people I do not know at all, most of which I ignore... Also, here is a thread on the MAPE that may be interesting, though I'll admit that I keep on saying the same things over and over... Dec 1 '21 at 20:47
• "It makes little sense to train a model to minimize the (R)MSE, then to assess its forecasts with a different error measure" - is this time-series specific? I was under the impression that in ML one often trains using a computationally/numerically well-behaved loss function but uses another, more interesting, error measure on the holdout data. (Crossing fingers that the two agree to some extent!) Dec 2 '21 at 17:12
• @AlbertoSantini: I would say that this is bad practice in either case. If my teacher tells my kids they should study lessons 1-5, and then the test is on lessons 6-10, the kids will be justifiably upset - so why should we do the analogue in statistics and ML? (Exception: I'm good with using a smooth approximation to a non-differentiable loss function.) I think this problem is more prevalent in classical forecasting, where most models implicitly yield expectation forecasts - in ML, it's more common to specify your own objective function. Dec 2 '21 at 17:24

Based on visual inspection your time series might have a discernable wave every 1 to 3 months. If you include longer period terms in your model this will create an extrapolated mean estimate and prediction interval that is more "wiggly." It's not critical, but it might improve the point predictions. As you currently have it the vast majority of the observations in the test set fall within the prediction intervals.

MAPE stands for mean absolute percentage error. It is the average multiplicative effect between each estimated mean and the observed outcome. RMSE stands for root mean squared error, i.e. standard deviation. While they both summarize the variability of the observations around the mean, they are not in the same scale so don't expect the values to be similar. I suggest using RMSE as this is the basis for how the model is fit to the data.

• Thank you very much for your answer Mr. Johnson. I understand totally. I increased the value fo K in harmonic regression as I learned from Mr. Hyndman as the line gets more wiggly by increasing K, but after a certain level the model fails the Ljung-Box test. Dec 1 '21 at 20:36