As @DikranMarsupial say, you need a nested validation procedure. In the inner e.g. cross validation, you do all the tuning of your model - that includes both choosing hyperparameters and model family.
In principle, you could also have a triply nested validation structure, with the innermost tuning the respective model family hyperparameters, the middle one choosing the model family and the outer as usual to obtain a generalization error estimate for the final model.
The disadvantage with this, however, is that splitting more often than necessary means that the data partitions become rather small and thus the whole procedure may become more unstable (small optimization/validation/test set mean uncertain performance estimates).
Nesting vs. cross validation or hold-out
Nesting is independent of the question what splitting scheme you employ at each level of the nested set-up. You can do cross validation at each level, single split at each level or any mixture you deem suitable for your task.
2 nested levels and both CV is what is often referred to as nested cross validation, 2 nested levels and both single split is equivalent to the famous train - validation [optimization] - test [verification] setup. Mixes are less common, but are a perfectly valid design choice as well.
If you have sufficient data so that single splits are a sensible option, you may also have sufficient data to do three such splits, i.e. work with 4 subsets of your data.
One thing you need to keep in mind, though, is: a single split in the optimization steps* you deprive yourself of a very easy and important means of checking whether your optimization is stable which cross validation (or doing several splits) provides.
* whether combined hyperparameter with model family or model family choice plus "normal" hyperparameter optimization
Triply nested vs. "normal" nested
This would be convenient in that it is easy to implement in a way that guards against accidental data leaks - and which I suspect is what you were originally after with your question:
estimate_generalization_error() which splits the data into test and train and on its train data calls
choose_model_family() which employs another internal split to guide the choice and calls and on its training split calls the various
optimize_model_*() which implement another internal split to optimize the usual hyperparameters for each model family (*), and on its training split calls the respective low-level model fitting function.
optimize_model_*() are an alternative to a combined tuning function that does the work of both in one split. Since both are training steps, it is allowed to combine them. If you do grid search for hyperparameter tuning, you can think of this as a sparse grid
with model family x all possible hyperparameters where evaluate only combinations that happen to exist (e.g. skip
mtry for SVM).
Or you look at the search space as a list of plausible hyperparamter combinations that you check out:
- logistic regression
- SVM with cost = 1, gamma = 10
- SVM with cost = 0.1, gamma = 100
- random forest with ...
to find the global optimum across model families and model family specific hyperparameters.
There is nothing special about
model_family - it is a hyperparameter for the final model like
gamma are for SVMs.
In order to wrap your head around the equivalence, consider optimizing gamma and cost for an SVM.
Method one: set up a grid or a list of all plausible cost; gamma combinations and search that for the optimum. This is the analogue to the "normal" nested approach.
- set up a list of all plausible cost values.
- for each cost value, optimize gamma.
- select the cost with best optimized gamma
This is the analogue to the triply nested approach.
In both cases, we can "flatten" the nested structure into a single loop iterating over a list or grid (I'm sorry, I lack the proper English terms - maybe someone can help?).
This is also vaguely similar to "flattening" a recursive structure into an iterative one [though the triply nested is not recursive, since we have different functions f(g(h()))].
This flattening approach potentially has the further advantage that it may be better suited to advanced optimization heuristics. As an example, consider moving from "select the observed optimum" to the one-standard-deviation rule. With the flattened approach, you can now look across model families which model is least complex not more than 1 sd worse than the observed optimum.