When does AIC lose its power to discriminate models?

Question

There are two simple questions at the end, but I think it is also useful to share the background that motivated them. It comes from this question on an unexpected forecast from the fully automatic methodology behind the forecast::ets function in R. The code, plot and forecast are given below for convenience:

library(forecast);options(scipen=999)
usage <- ts(scan('http://cl.ly/102L0j3o1p2m0m3p0t2o/usage'), frequency = 24)

plot(f1<-forecast(m1<-HoltWinters(usage), h = 168))
plot(f2<-forecast(m2<-ets(usage), h = 168));AIC(m2) #replication OK

plot(f3<-forecast(m3<-ets(usage,additive.only=TRUE), h = 168))
plot(f4<-forecast(m4<-ets(usage,additive.only=TRUE,damped=FALSE), h = 168))

Just by looking at the plot of the data it appears to me immediately that ARIMA(0,2,2) or ETS(AAN) will be among the best non-seasonal models (and their point forecasts will not differ much). Following the usual advice that the AIC of only a small set of potentially useful models should be compared, I can see no reason to consider multiplicative models here, nor can I see how a damped forecast will be useful for me. With this information in, the "best" ets model m4 and its associated forecast f4 is what was expected, but the process is not a fully automatic one.

With many time series, which I would not have time or desire to look at, I would hardly have a better option than to blindly use ets(data). The ets documentation page assures me that "The methodology is fully automatic. The only required argument for ets is the time series. The model is chosen automatically if not specified". The above example is not the first one to show that the methodology that works is at best semi-automatic and the fully automatic one may fail to work as expected even in simpler cases.

One way to rectify the problem is to consider n $\ge$ 1 models with suitably low values of AIC as equally supported by the data. I agree with RJH ("Although the model may not give the best AIC, it may give forecasts that are more useful to you.") that the most useful forecasts need not be those from the model with the smallest value of the AIC. But then we have the question of when the AIC is useful for selecting a model that forecasts best. It can be deduced from, for example, quotes in this question that if the AICs of all models considered differ by no more than some small number c, then the AIC loses its power to distinguish between these models and other factors/ideas must be considered. But how small is c?

Those working with Akaike (Y Sakamoto and M Ishiguro and G. Kitagawa) in the book entitled "Akaike Information Criterion statistics" (in the section entitled "Some remarks on the use of the AIC") mentioned c=1. The number c=2 is often mentioned (e.g. a quote from Brian Ripley can be added to those two linked to above). The number c=4 was mentioned by Chris Chatfield in one of his books. I have not seen anything explicit on the values of c greater than four, but this probably depends on the variability of data, sampling error of the deviance and related factors.

In the above example, AIC(m3,m4) suggests that the AIC of the model with a more useful forecast is greater than the AIC of the other model by "only" 78348.75-78337.22=11.52. Are there any formulae, guidelines or rules of thumb for useful values of c given data? Have values of c greater than four been mentioned in the literature?

Answer 1

The problem is that your data are not from anything like the family of ETS models. A quick look at the graph shows that the data involve something like a linear trend with a step change around $t=60$. There is no such model in the ETS class.

When you ask ets to find a suitable model, it can only find the best match within the class of ETS models. In this case, it chooses a damped trend because that step change makes the forecasts from a local linear model particularly bad during that short period.

The ARIMA class of models is probably better here as it includes models that are closer to what might have generated your data. For example, the following code does pretty well:

plot(f5 <- forecast(m5 <- auto.arima(usage)))

However, automatic forecasting is no substitute for looking at the data, thinking about where they came from, and constructing an appropriate model.

By the way, try dividing your data by 1e9 before doing any modelling. The sheer size of the numbers can cause problems.