Under certain conditions, AIC is an efficient model selection criterion. I understand this roughly as if AIC will tend to select the model that will yield the largest expected likelihood of a new data point from the same data generating process or population (among all models that we are selecting from). This makes AIC the preferred choice if the goal is prediction and the evaluation of predictions is the likelihood.

However, we do not always evaluate prediction accuracy by the likelihood. There are other means of evaluating predictions such as, say, mean squared error (MSE) or mean absolute error (MAE). Questions:

  1. Is AIC still the model selection method of choice if prediction accuracy is evaluated by these loss functions (MSE, MAE)?
  2. What could be a good counterexample, preferably among the well-known loss functions? I.e. what loss function would not favor AIC as the model selection criterion?
  3. How can we characterize the entirety of loss functions for evaluating prediction accuracy that are compatible with AIC being the method of choice for model selection?
  • $\begingroup$ For Q2. Is hinge loss considered "fair game"? Essentially any discontinuous loss function is reasonable candidate for a counterexample regarding a metric that is strongly based on a loglikelihood. $\endgroup$ – usεr11852 Sep 10 '19 at 18:45
  • $\begingroup$ @usεr11852, I do not have a strong opinion on this one. If you describe it in more detail and explain the intuition, it could make a good answer. $\endgroup$ – Richard Hardy Sep 10 '19 at 19:05
  • $\begingroup$ A somewhat related question: Equivalence of AIC and LOOCV under mismatched loss functions. $\endgroup$ – Richard Hardy Sep 10 '19 at 19:07

I think the answer to 1) should be "no", as there is no reason in general to expect that the model which minimizes the expected likelihood will also minimize the MSE, MAE, etc. One might even think of the case in which the likelihood is well defined and the MSE will diverge as the sample size increases (e.g. a distribution with no moments, such as the Cauchy).

I would think that AIC will still be a good method for model selection for any loss function which is a monotone function of expected likelihood, but that's probably a trivial remark that will not help you much.

| cite | improve this answer | |
  • $\begingroup$ You have answered all of the questions concisely, thank you. To complete the picture, it would be nice if you offered some illustration of loss functions that are monotone w.r.t. expected likelihood. $\endgroup$ – Richard Hardy Feb 11 at 14:27
  • $\begingroup$ Ping........... $\endgroup$ – Richard Hardy Jun 10 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.