time series forecasting using auto.arima and exponential smoothing

Question

I am working with workers’ remittance quarterly data for Bangladesh. Here I am doing time series forecasting using R. I am applying auto.arima model and exponential smoothing model. I want to compare between them to check which best fits the data and gives better forecast.

Here is the output:

fit1 <- auto.arima(lremit, d=1, D=NA, stationary=FALSE, + seasonal=TRUE,ic="aic",trace=TRUE, + allowdrift=FALSE,allowmean=TRUE)

Best model: ARIMA(2,1,3)(0,1,1)[4]

summary(fit1) Series: lremit ARIMA(2,1,3)(0,1,1)[4]
Coefficients: ar1 ar2 ma1 ma2 ma3 sma1 -0.5024 -0.1691 0.3940 0.1516 -0.1899 -0.9605 s.e. 0.6321 0.4860 0.6298 0.4465 0.1060 0.1098 sigma^2 estimated as 0.007314: log likelihood=135.59 AIC=-257.18 AICc=-256.3 BIC=-236.84

Training set error measures: ME RMSE MAE MPE MAPE MASE ACF1 Training set -0.003608938 0.08398593 0.06532171 -0.09985958 1.110818 0.4381367 -0.004851439

fit1 <- Arima(lremit,order=c (2,1,3),seasonal=c (0,1,1)) h11=plot(forecast(fit1,h=20)) h11 $mean Qtr1 Qtr2 Qtr3 Qtr4 2015 8.256047 8.283843 8.300686 8.341204 2016 8.372717 8.406483 8.413318 8.457855 2017 8.489041 8.522291 8.529440 8.573906 2018 8.605075 8.638346 8.645488 8.689954 2019 8.721124 8.754394 8.761536 8.806002

ETS

fit2<-ets(lremit) summary(fit2)

ETS(A,A,N)

Call: ets(y = lremit)

Smoothing parameters: alpha = 0.8594 beta = 1e-04

Initial states: l = 4.2135

sigma: 0.0858

 AIC     AICc      BIC 

12.20515 12.50145 23.97172

Training set error measures: ME RMSE MAE MPE MAPE MASE ACF1 Training set -7.229862e-05 0.08579429 0.06800397 -0.01942594 1.169213 0.4561276 -0.002900248

It is my first work using R and I am facing problems regarding this. They are:

1. auto.arima output shows seasonality in every 4th quarter, but exponential smoothing shows non seasonality, what is the interpretation of this contradictory result?
2. How can I compare between them, what is the proper measure?
3. What is the command for in sample forecast in auto.arima? If I write h=0, then it shows error
4. Where can I find elaborate interpretation of auto.arima and exponential smoothing output and about the comparison?
5. Which error measure should I prefer like ME, MAPE, RMSE etc. as they are almost same for the two models?
6. In case of auto.arima it shows same output for allowing drift or no drift

Answer 1

1. Seasonality is probably not very strong. Different algorithms will give different results, unless seasonality is glaringly obvious.

2. The best measure is always to compare forecast accuracy on a holdout set: hold back the last $n$ observations, fit your models to all other observations, forecast into the last $n$ time periods with both models, then compare forecast accuracy using your error measure of choice (see 5 below).

3. Yes, this is a common complaint. I don't think there is an easy way to get the in-sample fit. But you can get the residuals: auto.arima(WWWusage)$residuals. Best to look into the code of auto.arima() to see whether you need to add or subtract them from the original series to get the fit. I'd say you have to subtract ("actuals=model+residuals"), but better check.

4. I recommend a good forecasting textbook. This is a very good start. Otherwise, read through the help pages.

5. The appropriate error measure will depend on your personal loss function. Is your pain symmetric, and will it increase more strongly with larger errors? Then use MSE. Is your pain proportional to absolute errors? Then use MAE. Best to look at multiple error measures.

   One tip: averaging forecasts will usually improve accuracy. Consider taking the average of your two models' forecasts per future time bucket.

6. auto.arima() apparently fits no drift, even if you allow it.