# As you drop variables, can AIC or BIC go up and then down?

I have some potential spline models and I'm trying to use AIC or BIC to choose variables. I'm seeing that AIC is lower when I use all variables than if I exclude any one or two. However, if I exclude three variables, then AIC is lower than if I include all variables. Is this theoretically possible or am I potentially doing something wrong? My concern is that if I do backwards selection with AIC, the optimal model chosen will be all variables since incrementally dropping one will not reduce AIC.

Sorry I can't post a reproducible example as I have a quite complex spline model and the data is proprietary to my company.

• Relevant Jul 14 '18 at 0:31

Remember that $$\mathrm{AIC} = 2k - 2 \log L$$ where $k$ is the number of parameters and $L$ is the likelihood. As you remove features, both $k$ and $L$ decrease, so the $2k$ term decreases and the $-2 \log L$ term increases. Each of those terms is monotonic as you remove features, but their sum need not be, as they change at different rates.