0
$\begingroup$

Let's say you have a twice differenced time series. The AIC for two potential models are 2000 and 2000.1. As a rule of thumb you usually choose the one with the lower AIC, but if the model with AIC 2000.1 is a slightly simpler model would you be more inclined to choose it since both AIC's are within 2 units of each other?

$\endgroup$
1
1
$\begingroup$

Under some assumptions, AIC is an optimal model selection criterion. It strikes an optimal balance between goodness of fit and model complexity. Under these assumptions, it does not make sense to choose a model with a higher AIC value, even if the difference between the AIC values of the models under consideration is (very) small.

However, when the difference is small, you would also expect a small difference in performance. If you have to choose a single model, you would still go with the one that minimizes the AIC. But if you have an option to choose more than one model and do some sort of model averaging, a small difference between the best models would suggest both of them deserve to be part of the model combination. (It is commonly observed in forecasting that model combinations do well relative to the individual models.)

$\endgroup$
6
  • $\begingroup$ I had seen on a textbook (can't remember which) that once they were within 2 units it would be common to select the simpler model. But it makes sense what you say that the AIC already accounts for this impacts it negatively for more complex models. Good idea about including both models also thank you for you insight. $\endgroup$ – Craig Apr 18 '20 at 17:28
  • $\begingroup$ I just came across a question on AIC values which might back up my initial assumption of AIC's falling within 2 units of each other. There might not be a set answer to this by the looks of it. The question was posted here: stats.stackexchange.com/questions/232465/… $\endgroup$ – Craig Apr 18 '20 at 17:48
  • $\begingroup$ @Craig, thank you for the link. There are opinions and there are facts. What I stated are facts. Then we can have opinions on how well the assumptions are satisfied (some of them can be tested, so there we would again have facts) and how relevant the specific type of optimality is to a particular user. I would just like to point to my first paragraph. I see no principled reason for choosing a model with a higher value of AIC. On the other hand, when AIC values are close, the error one will make will likely be small. $\endgroup$ – Richard Hardy Apr 18 '20 at 18:37
  • $\begingroup$ Is it a fact that very small differences in AIC actually suggest a meaningful better model? All the discussions I have read of that suggest this is not so. $\endgroup$ – user54285 Apr 18 '20 at 22:46
  • $\begingroup$ @user54285, "meaningful" is not a statistical term. But you could embed this in a problem of utility maximization and see how different the expected utilities are. It is certainly possible they are "meaningfully" different. E.g. it is possible that you expect to earn \$2000 from using one model and \$1000 from using another. The difference is "meaningful" to me. We can up the stakes until it becomes "meaningful" to you. $\endgroup$ – Richard Hardy Apr 19 '20 at 7:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.