I've got some classification data to model and a selection of possible learning algorithms to use, varying from simple logistic regression, to lasso, neural networks, and random forests.
What is the best way of selecting the best learning algorithm with its most effective parameter choice, and then providing a measure of unbiased generalisation for the chosen model?
I have 3 goals:
- Establish the best learning algorithm
- Select a final model
- Provide an estimate of generalising ability
I've seen numerous posts on this site by @Dikran Marsupial and @cbeleites which state that nested CV is the best approach for model evaluation (#3) when you have a single learning algorithm with hyperparameters to optimise. I.e. if I knew I were going to use a Neural Net I'd run Nested CV with the inner loop selecting the best choice of # of hidden nodes, and the outer loop measuring this accuracy. I'd then use the outer CV score as a measure of unbiased generalisation performance and form my final model using a single layer of CV to select params on the whole data.
Say I had a Neural Network with the hyper parameter h representing the number of hidden nodes with 3 possible values, and a Random Forest with the hyper parameter t indicating the number of trees to include in the forest, with 4 possible values.
To select a final model (goal 2), I can run a single layer of CV with both learning algorithms and their hyperparameters as free parameters (as Marc suggested) to be optimised, resulting in 7 averaged CV scores from which I select the most accurate combination for my final model building.
For goal 3 I can run nested CV to produce an unbiased estimate of the generalising ability of my overall method, where at each outer fold I select the single most accurate combination of learning algorithm and hyper parameter from the 7 available (i.e. RF with t = 100 at K=1, NN with h=10 at K=2...).
However what would be the most effective approach for goal 1, where I want to say that overall, either Random Forests or Neural Networks are the most effective learning algorithms for this particular dataset?
Could I just select the optimal hyper parameters for each learning algorithm from the single layer of CV used in the model selection (goal 2)? Say the most accurate value of t is 100, and the optimal h = 5, then could I just compare the CV accuracies of Random Forests with t=100 to Neural Networks with h=5? Intuitively this would seem biased.
Or would it be better to run a nested CV approach as in goal 3, but rather than selecting the best overall combination of learning algorithm and hyper parameter at each outer fold from an inner CV, select the best parameter value per learning algorithm? I.e. at outer fold K=1 select the best value of t and h and test these on outer validation fold and repeat for K <= 10 resulting in a CV score for both RF and NN which both have had their parameters tuned fairly.