8
$\begingroup$

I have a doubt related to using AIC for model selection - in which case it may not recommend the true best predictive model (based on my understanding). I understand AIC has 2 terms - goodness of fit (which can be obtained my finding error on model training dataset) and complexity term (2*no. of parameters in model). I discuss the case below:

I have 2 models - 1st model is non-parametric model that interpolates each data point and overfits, hence no. of parameters (K) is same as sample size (lets say 500). SSE on train set (goodness of fit) is very good, say 1e-4 (as it overfits the trained dataset). Its calculated AIC value (using formula, n*ln(SSE/n)+2K) would be -6712. The second model is a parametric model (2nd order polynomial regressive model) with 6 parameters. Its goodness of fit is not as good as non-parametric model with SSE being 1e-1. Its calculated AIC would be -4246.

Based on delta(AIC), we would select model 1, but we know that the 1st model overfits the data and hence would not generalize well on new data.

So, how do we use AIC in such cases when a model overfits data but the complexity term does not penalize it well enough to reject it among others. Does this case imply we can not use AIC for differentiating between parametric and non-parametric models?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Perhaps the nonparametric model overfits, but how badly does the parametric model underfit? This will depend on the data generating process and signal-to-noise ratio. Without additional information, you cannot decisively say that AIC made a bad choice here. There other problem I suspect is that the size of the nonparametric model grows 1 to 1 with the size of the dataset, something that is not within the conditions required for AIC to be (asymptotically) optimal for prediction. $\endgroup$
    – Richard Hardy
    Commented Feb 24, 2020 at 8:25
  • $\begingroup$ Thanks alot @RichardHardy. I get this, but what if the parametric model behaves reasonably well (if I use train-test split and see its predictive performance, it gives better predictions than non-parametric overfit model as the latter captured errors as well). Can I say this - "interpolative" models cannot be used to compare with other parametric models, as parameters in former are not estimated by MLE (as their parameters are unique to exactly map output response and model predictions)? For moderate sample size, interpolative model would give near perfect gof, hence AIC would likely be small. $\endgroup$
    – Maaz
    Commented Feb 25, 2020 at 4:31
  • $\begingroup$ It is all relative. The fit would be good, but the number of parameters would be large, so AIC would not be all too small. Regarding estimation by MLE, I think it is a valid point. $\endgroup$
    – Richard Hardy
    Commented Feb 25, 2020 at 6:52

1 Answer 1

Reset to default
9
$\begingroup$

AIC can most definitely select an overfit model, because you e.g.

  1. only assess overfit models (so one of those gets selected), or
  2. you offer up an overfit model vs. an inappropriate model (seems to be your example)/a very overfit model/an underfit model, or
  3. you compare more than one model via AIC (the more models the worse this gets) and by testing several you end up overfitting via the model selection.

While AIC attempts to balance fit to the training data vs. model complexity, there is nothing inherent to it that would provide a guarantee that model selection would result in an non-overfit model (of course, it penalizes model complexity more than if you selected solely on fit to the training data, so in that sense overfitting ought to be a bit more avoided). In fact (see point 3 above), the very fact of doing model selection involves the potential for overfitting to the data on which you select the model and model averaging (and various other approaches) have been proposed to avoid this issue (see e.g. Model Selection and Multimodel Inference by Burnham and Anderson).

Improve this answer
$\endgroup$
5
  • $\begingroup$ A really extreme case in which AIC can select an overfit model: if you have normal error terms, perfectly fitting each data point (i.e., linear regression with more covariates than data points) leads to an unbounded likelihood, which leads to an unbounded AIC score, despite (most likely) being hugely overfit! $\endgroup$
    – Cliff AB
    Commented Feb 24, 2020 at 17:36
  • $\begingroup$ Thanks @Bjorn! Dont you think that its better to go for approach like crossval or train-test validation etc. to select a good predictive model? Can you kindly help me explain why would one go for AIC/BIC metrics if they can lead to poor recommendations? $\endgroup$
    – Maaz
    Commented Feb 25, 2020 at 4:36
  • $\begingroup$ @Maaz, note that under some (I guess mild) assumptions, AIC/BIC are asymptotically equivalent to cross validation. So if you can tring AIC/BIC, you can also trick cross validation. $\endgroup$
    – Richard Hardy
    Commented Feb 25, 2020 at 6:54
  • $\begingroup$ Firstly, AIC is super fast to calculate and when the answer is super clear, it may be no need to do something more time consuming. Some of the same limitations also apply to CV. $\endgroup$
    – Björn
    Commented Feb 25, 2020 at 7:04
  • $\begingroup$ Alright I agree with both these points. I guess then, it might boil down to applicability of AIC for interpolative models due to not fulfilling MLE parameter estimation requirement! Thanks $\endgroup$
    – Maaz
    Commented Feb 25, 2020 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.