1
$\begingroup$

When tuning regularized models, two techniques appear to be especially popular at the moment:

  1. Cross-validation performance on train & validation splits (the third, test/holdout set is not used in the tuning step or model selection, but only to report an estimate of the "true performance" in the wild.), see below:

Cross Validation Schema: https://miro.medium.com/max/640/1*PpFIzhckhq9qFAEbwrXY2g.png

  1. Information Criterion (AIC/BIC/MDL etc.)

Here's a lovely example in the sklearn documentation detailing the selection of alpha for a Lasso regression via the two methods mentioned above.

Both techniques have their pros and cons. IC's are generally less computationally intense and less data-hungry than Cross-Validation. Cross-Validation, on the other hand, provides us with an estimate of the uncertainty in the model (an incredibly important concept to quantify in certain fields).

In cases where inference, not accuracy, is of particular import, the use of Information Criteria (IC) seems a logical one as they often incorporate the concept of "sparsity"/"complexity" in the scoring of a given model in addition to an accuracy metric. Sparser models are desirable for many reasons, but especially with inference as an objective.

My question is: Can we use IC's as a scoring function during cross-validation?

Can ICs and Cross-Validation be combined or is the computation over multiple splits some sort of "IC hacking"?

Can we - During hyperparameter tuning using cross-validation:

  1. Compute an IC of choice on the training data* for each split (4 green training folds per split in the diagram above).
  2. Pool the ICs of the splits using some aggregation (usually the mean).
  3. Have our tuners optimize hyperparameters based on this aggregate? - effectively, perform model comparisons using this aggregated quantity.

Or, when using IC's to tune parameters, are we only allowed to compute these quantities on the training part of a simple train/test split?

I'm asking a similar question to this post, but asking if I can calculate ICs on the in-sample data during cross-validation.

*We would not compute this quantity on the validation sets (blue blocks in the diagram), as that is akin to computing on the test set if we were doing a simple train/test split. Nice explanation here. ###############################################################

Why would we want to do this?

I'm considering ICs as scoring functions during hyperparameter tuning. Particularly, I view them as scoring functions which help strike a balance for parsimony. I'm interested in using them with other metrics (say the Matthews correlation coefficient for classification) during cross-validation. As a voice for "sparsity", the inclusion of an IC may provide a softer "complexity penalty" than simply returning the count of non-zero coefficients directly. Because they also often consider "fit", I may be able to eliminate yet another objective term from my optimizer.

I guess what I'm looking for is some guidance on scoring functions or combinations thereof, that help optimize for parsimony. I see IC's as a group of functions that are widely accepted (and well documented) balances of fit and complexity. I want to understand what the effect of computing these during cross validation, aggregating, and using as a metric of parsimony may have.

Thanks all!

$\endgroup$
1
  • $\begingroup$ Thanks @CagdasOzgenc for responding. Based on your comment I rephrased my question a bit to try and clarify it. If you get a chance, please take a look. $\endgroup$
    – FiddleBat
    Commented Nov 13, 2022 at 22:14

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.