There are two simple questions at the end, but I think it is also useful to share the background that motivated them. It comes from this question on an unexpected forecast from the fully automatic methodology behind the forecast::ets function in R. The code, plot and forecast are given below for convenience:
library(forecast);options(scipen=999) usage <- ts(scan('http://cl.ly/102L0j3o1p2m0m3p0t2o/usage'), frequency = 24) plot(f1<-forecast(m1<-HoltWinters(usage), h = 168)) plot(f2<-forecast(m2<-ets(usage), h = 168));AIC(m2) #replication OK
plot(f3<-forecast(m3<-ets(usage,additive.only=TRUE), h = 168)) plot(f4<-forecast(m4<-ets(usage,additive.only=TRUE,damped=FALSE), h = 168))
Just by looking at the plot of the data it appears to me immediately that ARIMA(0,2,2) or ETS(AAN) will be among the best non-seasonal models (and their point forecasts will not differ much). Following the usual advice that the AIC of only a small set of potentially useful models should be compared, I can see no reason to consider multiplicative models here, nor can I see how a damped forecast will be useful for me. With this information in, the "best" ets model m4 and its associated forecast f4 is what was expected, but the process is not a fully automatic one.
With many time series, which I would not have time or desire to look at, I would hardly have a better option than to blindly use ets(data). The ets documentation page assures me that "The methodology is fully automatic. The only required argument for ets is the time series. The model is chosen automatically if not specified". The above example is not the first one to show that the methodology that works is at best semi-automatic and the fully automatic one may fail to work as expected even in simpler cases.
One way to rectify the problem is to consider n $\ge$ 1 models with suitably low values of AIC as equally supported by the data. I agree with RJH ("Although the model may not give the best AIC, it may give forecasts that are more useful to you.") that the most useful forecasts need not be those from the model with the smallest value of the AIC. But then we have the question of when the AIC is useful for selecting a model that forecasts best. It can be deduced from, for example, quotes in this question that if the AICs of all models considered differ by no more than some small number c, then the AIC loses its power to distinguish between these models and other factors/ideas must be considered. But how small is c?
Those working with Akaike (Y Sakamoto and M Ishiguro and G. Kitagawa) in the book entitled "Akaike Information Criterion statistics" (in the section entitled "Some remarks on the use of the AIC") mentioned c=1. The number c=2 is often mentioned (e.g. a quote from Brian Ripley can be added to those two linked to above). The number c=4 was mentioned by Chris Chatfield in one of his books. I have not seen anything explicit on the values of c greater than four, but this probably depends on the variability of data, sampling error of the deviance and related factors.
In the above example, AIC(m3,m4) suggests that the AIC of the model with a more useful forecast is greater than the AIC of the other model by "only" 78348.75-78337.22=11.52. Are there any formulae, guidelines or rules of thumb for useful values of c given data? Have values of c greater than four been mentioned in the literature?