3
$\begingroup$

This is an extension of a previous question: How to avoid overfitting bias when both hyperparameter tuning and model selecting? ...which provided some options for the question at hand, but now I would like to pivot to knowing what is accepted practice or rule of thumb.

In short, say we do hyperparameter tuning on multiple ML model families. The following selection step of choosing the model family itself provides another opportunity for optimistic bias. This could be resolved by some of the strategies noted in the link above.

Noting the previous discussion, are there accepted rules of thumb (or research) on when said strategies are important? For instance, if just optimizing two model families, is it generally safe to ignore the concern and pick the model family in the train split score (or perhaps even the test split)? Or is there a certain n number of model families at which this becomes a danger and tripple-nesting or gridsearch modifications of some kind is needed?

$\endgroup$

2 Answers 2

1
$\begingroup$

Model selection can often be seen - in a broad sense - as part of hyperparameter tuning.

An example: let's say that we have to solve a simple regression problem, and we want to use some sort of linear model. In this scenario we could choose linear models with or without a polynomial expansion, with or without a L1 or L2 regularization term.

One could see the problem as follows:
- A model selection between L1 regularized, L2 regularized, and not regularized models
- Hyperparameter tuning to define the order of polynomial and (if present) regularization terms

Alternatively, it can be seen as just one large model (like an Elastic Net) with polynomial expansion, and everything becomes a hyperparameter.

This example is particularly trivial, because hyperparameter tuning and model selection are directly linked by the regularization coefficient (if we put them to 0, we go into the non-regularized model). However this can always be done, and model selection can be seen as part of hyperparameter tuning, with the set of hyperparameters being conditional on the choice of the first hyperparameter (the model).

This might sound weird, but conditional hyperparameters are very common: for example, the hyperparameter number of units in the 3rd layer of my Neural Net is conditional to the hyperparameter depth of my neural network being larger than 2.

So finally what's the best practice?
I would say that it depends on the amount of data you have and the amount of hyperparameters you use. The less data and the more hyperparameters, the more bias you will have. If you have enough data, doing a nested cross validation is most likely overkill and you should stick to a flat CV, as mentioned in the post above.
If however you are testing a huge amount of parameter configurations or different models, it might be worth either using nested cross validation, or using a separate validation set to check the results of your cross validation. This can mean either doing hyperparameter tuning via cross validation, and model selection on a separate set, or even simply re-scoring your best performing models and configurations after you filtered them via cross val.

$\endgroup$
3
  • $\begingroup$ This would be my intuition also, though I wish we could get some more rules of thumb on concepts like "a lot", "many" etc. Perhaps that is not possible. Can you clarify the last line though? ("or even simply re-scoring your best performing models and configurations after you filtered them via cross val") $\endgroup$
    – Josh
    Commented Nov 25, 2020 at 18:47
  • $\begingroup$ I do not think it is really possible to quantify these, but maybe some research could shed some light on the orders of magnitude. $\endgroup$
    – Davide ND
    Commented Nov 26, 2020 at 10:02
  • $\begingroup$ Wrt the last line: if I test thousands of configurations (or models) I might not blindly trust the flat CV. What I do is use the results of the flat CV to rank my configurations/models and keep a top percentile (say the best 50, or 10, or those whose CV score is significantly higher). Then you score these on a separate, unbiased validation set, and take the best from there (or bests, if even better you want to average or stack them). This allows you to have a better scoring that avoids that optimistic bias, and at the same time is less burdensome than nested CV. $\endgroup$
    – Davide ND
    Commented Nov 26, 2020 at 10:10
0
$\begingroup$

I wrote a paper (with a co-author) on these topics (nested vs non nested cross-validation).

Please see it at https://arxiv.org/abs/1809.09446

TLDR: for practical purposes we advise against using nested cross validation. Although there is indeed a positive/overfitting bias in performing a flat search (as opposed to nested) the difference is below what we consider a practical threshold of irrelevance (which is compatible with others proposal for this threshold).

But this is a practical proposal, when your goal is to select the best model+hyperparameters for your problem. If you have a scientific problem of showing that your algorithm is better than the competition, you should perform the nested CV.

$\endgroup$
1
  • $\begingroup$ This is interesting and I have had skepticism against nested cv. I must take some time to digest it though $\endgroup$
    – Josh
    Commented Nov 25, 2020 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.