# Model selection when AIC scores are very close

Let's say you have a twice differenced time series. The AIC for two potential models are 2000 and 2000.1. As a rule of thumb you usually choose the one with the lower AIC, but if the model with AIC 2000.1 is a slightly simpler model would you be more inclined to choose it since both AIC's are within 2 units of each other?

• Related. Apr 18, 2020 at 17:18

## 1 Answer

Under some assumptions, AIC is an optimal model selection criterion. It strikes an optimal balance between goodness of fit and model complexity. Under these assumptions, it does not make sense to choose a model with a higher AIC value, even if the difference between the AIC values of the models under consideration is (very) small.

However, when the difference is small, you would also expect a small difference in performance. If you have to choose a single model, you would still go with the one that minimizes the AIC. But if you have an option to choose more than one model and do some sort of model averaging, a small difference between the best models would suggest both of them deserve to be part of the model combination. (It is commonly observed in forecasting that model combinations do well relative to the individual models.)

• I had seen on a textbook (can't remember which) that once they were within 2 units it would be common to select the simpler model. But it makes sense what you say that the AIC already accounts for this impacts it negatively for more complex models. Good idea about including both models also thank you for you insight. Apr 18, 2020 at 17:28
• I just came across a question on AIC values which might back up my initial assumption of AIC's falling within 2 units of each other. There might not be a set answer to this by the looks of it. The question was posted here: stats.stackexchange.com/questions/232465/… Apr 18, 2020 at 17:48
• @Craig, thank you for the link. There are opinions and there are facts. What I stated are facts. Then we can have opinions on how well the assumptions are satisfied (some of them can be tested, so there we would again have facts) and how relevant the specific type of optimality is to a particular user. I would just like to point to my first paragraph. I see no principled reason for choosing a model with a higher value of AIC. On the other hand, when AIC values are close, the error one will make will likely be small. Apr 18, 2020 at 18:37
• Is it a fact that very small differences in AIC actually suggest a meaningful better model? All the discussions I have read of that suggest this is not so. Apr 18, 2020 at 22:46
• @user54285, "meaningful" is not a statistical term. But you could embed this in a problem of utility maximization and see how different the expected utilities are. It is certainly possible they are "meaningfully" different. E.g. it is possible that you expect to earn \$2000 from using one model and \$1000 from using another. The difference is "meaningful" to me. We can up the stakes until it becomes "meaningful" to you. Apr 19, 2020 at 7:07