How to choose between different learning algorithms

Question

I've got some classification data to model and a selection of possible learning algorithms to use, varying from simple logistic regression, to lasso, neural networks, and random forests.

What is the best way of selecting the best learning algorithm with its most effective parameter choice, and then providing a measure of unbiased generalisation for the chosen model?

I have 3 goals:

1. Establish the best learning algorithm
2. Select a final model
3. Provide an estimate of generalising ability

I've seen numerous posts on this site by @Dikran Marsupial and @cbeleites which state that nested CV is the best approach for model evaluation (#3) when you have a single learning algorithm with hyperparameters to optimise. I.e. if I knew I were going to use a Neural Net I'd run Nested CV with the inner loop selecting the best choice of # of hidden nodes, and the outer loop measuring this accuracy. I'd then use the outer CV score as a measure of unbiased generalisation performance and form my final model using a single layer of CV to select params on the whole data.

Say I had a Neural Network with the hyper parameter h representing the number of hidden nodes with 3 possible values, and a Random Forest with the hyper parameter t indicating the number of trees to include in the forest, with 4 possible values.

To select a final model (goal 2), I can run a single layer of CV with both learning algorithms and their hyperparameters as free parameters (as Marc suggested) to be optimised, resulting in 7 averaged CV scores from which I select the most accurate combination for my final model building.

For goal 3 I can run nested CV to produce an unbiased estimate of the generalising ability of my overall method, where at each outer fold I select the single most accurate combination of learning algorithm and hyper parameter from the 7 available (i.e. RF with t = 100 at K=1, NN with h=10 at K=2...).

However what would be the most effective approach for goal 1, where I want to say that overall, either Random Forests or Neural Networks are the most effective learning algorithms for this particular dataset?

Could I just select the optimal hyper parameters for each learning algorithm from the single layer of CV used in the model selection (goal 2)? Say the most accurate value of t is 100, and the optimal h = 5, then could I just compare the CV accuracies of Random Forests with t=100 to Neural Networks with h=5? Intuitively this would seem biased.

Or would it be better to run a nested CV approach as in goal 3, but rather than selecting the best overall combination of learning algorithm and hyper parameter at each outer fold from an inner CV, select the best parameter value per learning algorithm? I.e. at outer fold K=1 select the best value of t and h and test these on outer validation fold and repeat for K <= 10 resulting in a CV score for both RF and NN which both have had their parameters tuned fairly.

Answer 1

You can do that in nested cross-validation too, except that in this case you are selecting both a learning method and its hyperparameterization. The result would be a correct performance estimate of learning a model from your data, rather than learning a given model type. On a conceptual level, you can think of both learning algorithms and their hyperparameterizations as some sort of free parameters in your full learning setup.

Automatically determining the right learning approach is becoming quite easy via the use of libraries that can optimize both algorithms and their hyperparameters jointly. One of the libraries that allow you to do these things is Optunity, for which I am the lead developer.

You can find an example of how to optimize both algorithm and hyperparameterization here. All the code you see there would be wrapped by an outer cross-validation procedure to create a nested cross-validation setup.