On p. 34 of his PRNN Brian Ripley comments that "The AIC was named by Akaike (1974) as 'An Information Criterion' although it seems commonly believed that the A stands for Akaike". Indeed, when introducing the AIC statistic, Akaike (1974, p.719) explains that
"IC stands for information criterion and A is added so that similar statistics, BIC, DIC etc may follow".
Considering this quotation as a prediction made in 1974, it is interesting to note that in just four years two types of the BIC statistic (Bayesian IC) were proposed by Akaike (1977, 1978) and Schwarz (1978). It took Spiegelhalter et al. (2002) much longer to come up with DIC (Deviance IC). While the appearance of the CIC criterion was not predicted by Akaike (1974), it would be naive to believe that it was never contemplated. It was proposed by Carlos C. Rodriguez in 2005. (Note that R. Tibshirani and K. Knight's CIC (Covariance Inflation Criterion) is a different thing.)
I knew that EIC (Empirical IC) was proposed by people of Monash University in around 2003. I've just discovered the Focused Information Criterion (FIC). Some books refer to Hannan and Quinn IC as HIC, see e.g. this one). I know there should be GIC (Generalised IC) and I've just discovered the Information Investing Criterion (IIC). There is NIC, TIC and more.
I think I could possibly cover the rest of the alphabet, so I am not asking where the sequence AIC,BIC,CIC,DIC,EIC,FIC,GIC,HIC,IIC,... stops, or what letters of the alphabet have not been used or been used at least twice (e.g. the E in EIC can stand for either Extended or Empirical). My question is simpler and I hope more practically useful. Can I use those statistics interchangeably, ignoring the specific assumptions they were derived under, the specific situations they were meant to be applicable in, and so on?
This question is partly motivated by Burnham & Anderson (2001) writing that:
...the comparison of AIC and BIC model selection ought to be based on their performance properties such as mean square error for parameter estimation (includes prediction) and confidence interval coverage: tapering effects or not, goodness-of-fit issues, derivation of theory is irrelevant as it can be frequentist or Bayes.
Chapter 7 of Hyndman et al.'s monograph on exponential smoothing appears to follow the B-A advice when looking into how well the five alternative ICs (AIC, BIC, AICc, HQIC, LEIC) perform in selecting the model that forecasts best (as measured by a newly proposed error measure called MASE) to conclude that the AIC was a better alternative more often. (The HQIC was reported as the best model selector just once.)
I am not sure what is the useful purpose of the research exercises that implicitly treat all ICc as though they were derived to answer one and the same question under equivalent sets of assumptions. In particular, I am not sure how it is useful to investigate the predictive performance of the consistent criterion for determining the order of an autoregression (that Hannan and Quinn derived for ergodic stationary sequences) by using it in the context of the non-stationary exponentially smoothing models described and analysed in the monograph by Hyndman et al. Am I missing something here?
Akaike, H. (1974), A new look at the statistical model identification, IEEE Transactions on Automatic Control 19(6), 716-723.
Akaike, H. (1977), On entropy maximization principle, in P. R. Krishnaiah, ed., Applications of statistics, Vol. 27, Amsterdam: North Holland, pp. 27-41.
Akaike, H. (1978), A Bayesian analysis of the minimum AIC procedure, Annals of the Institute of Statistical Mathematics 30(1), 9-14.
Burnham, K. P. & Anderson, D. R. (2001) Kullback–Leibler information as a basis for strong inference in ecological studies, Wildlife Research 28, 111-119
Hyndman, R. J., Koehler, A. B., Ord, J. K. & Snyder, R. D. Forecasting with exponential smoothing: the state space approach. New York: Springer, 2008
Ripley, B.D. Pattern Recognition and Neural Networks. Cambridge: Cambridge University Press, 1996
Schwarz, G. (1978), Estimating the dimension of a model, Annals of Statistics 6(2), 461-464.
Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002), Bayesian measures of model complexity and t (with discussion), Journal of the Royal Statistical Society. Series B (Statistical Methodology) 64(4), 583-639.