Using AIC/BIC within cross-validation for likelihood based loss functions

Question

For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose the "best" number of classes in the mixture on the lines of the discussion here. For a given loss function and a classes $c$, CV calculates: $$ CV(c) = \dfrac{1}{n} \sum Lo(f, \hat{f}_c)\,, $$ where $Lo(f, \hat{f}_c)$ is the loss from fitting an MLE-based $c$ component model to the data. The natural loss function to use is the negative log-likelihood. So that the final CV estimates for fitting a model with $c$ classes are $$ CV(c) = \dfrac{1}{n} \sum_{i=1}^{n} \left(-\log L(x_{-i},y_{-i}|c)\right)\,.$$

where $L(x_{-i},y_{-i}|c)$ is the estimated likelihood from the held-out data using MLE estimates. If parsimonious models are preferred over over-fitting, does it make sense to use AIC/BIC or other penalized likelihood based loss functions within cross-validation, so that: $$ CV(c) = \dfrac{1}{n} \sum_{i=1}^{n} \left(-2\log L(x_{-i},y_{-i}|c) + 2p\right)\,?$$

Of course, we could use AIC/BIC as stand-alone model selection criterions. But I haven't seen them used within cross-validation. Is there a reason why this doesn't many sense?

Answer 1

Since you are evaluating the error on the holdout subsample, there is no reason to adjust twice the estimated negative log-likelihood for overfitting by adding $2p$. This adjustment would introduce a bias in your estimated twice the negative log-likelihood without adding any benefits to the estimate. Hence, I would not do as you do.

However, you could evaluate the likelihood on all data (training+holdout) instead of just the holdout data. Then adding the penalty to adjust for overfitting would make sense. You would no longer need to split the data into training and holdout and do cross validation, which would save computations.

Regarding

If parsimonious models are preferred over over-fitting

If you are after prediction accuracy (defined in terms of the likelihood of a data point), you would actually prefer the model that yields the maximum likelihood on a test sample (yet unobserved). This is why you would either use AIC in place of the likelihood for all data or use likelihood on the holdout samples in cross validation. This should make sense unless parsimony is a goal in itself (in addition to or in place of prediction accuracy). If so, you may formally define your objective to reflect parsimony (and perhaps prediction accuracy) and ask a question referring to that particular objective.

---

Update: Regarding the updated question, a loss function that depends not only on the prediction accuracy (as measured e.g. by the likelihood but potentially also by expected absolute error, expected squared error and the like) but also on the number of parameters is one that explicitly penalizes model complexity. If your goal is indeed to have a parsimonious model and you do not mind sacrificing some prediction accuracy, then you may well use such a loss function instead of the likelihood. AIC and BIC would be special cases of such a setup.

Regarding the first comment, the rationale for having a penalized loss function used for training is not the same as that for having a penalized loss function used for performance evaluation out of sample.

• The former typically still targets accurate prediction; penalization works to compensate for overfitting and is intended to achieve parameter estimates that yield accurate predictions.
• The latter explicitly targets parsimony in addition to prediction accuracy. There is an explicit conflict between the two, requiring a compromise solution with somewhat good accuracy and some parsimony.