If I would like to choose between different classifiers, e.g. support vector machines (SVM) and boosted trees, based on their generalization performance, can I do this in the same way as I would do model selection for just one classifier, i.e. hyper-parameter tuning of the cost parameter of SVM's?
I'll describe here is what I've learned about cross-validation and model selection so far, based on my reading of the elements of statistical learning, CrossValidated, the caret package and scikit. If I've misunderstood anything at this level, then I should start studying the basics again before considering model comparison.
In an ideal scenario we would split our dataset into three parts, a training, validation and test set. We would fit our SVM with a certain hyper-parameter value to the training set and evaluate its generalization error on the validation set, because simply looking at the training set error would be an over-optimistic and biased estimate. Then, we would repeat this for another hyper-parameter value, and so on and so forth until we have done this for all values we are interested in (e.g. using a grid search). Then we would select the model that shows the highest accuracy (or a different performance metric we are interested in). However, at this point, the accuracy of this best model is no longer representative of the generalization error. The reason is that by optimising for the hyper-parameter, information has leaked into the validation dataset that was used to estimate the performance. That is why we set out the test set at the very start, because we can use the best model to predict on this test set and get an unbiased estimate. ( Optionally the best performing model, as evaluated by the validation set, can be built on the training+validation set together, before assessing its performance on the test set. )
Now, if data is sparse, we can either use single or nested cross-validation to be more efficient.
Single CV: Instead of using a training, validation and test set, we only use a training and test set. Cross-validation on the training set gives an unbiased estimate of the generalization error, without the need to set aside an independent validation set. By repeating this CV for various hyper-parameter values, we can perform model selection. This is often called hyper-parameter tuning or a grid-search. After selecting the best hyper-parameter value, this model is fit to the entire training set and then evaluated on the testing set, to estimate its generalization error. (The fitting the selected model on the entire training set-thing is what I believe the R package caret is doing: [section 4, end of first paragraph(http://www.jstatsoft.org/v28/i05)).
Nested CV: To be even more efficient, we can use a single dataset and use an inner-CV to select the best model (tuned hyper-parameters) and an outer-CV to estimate its generalization error. (I am not sure if the final model is obtained by fitting the selected model to the entire set or not, this post suggests you should.)
Now, what if I'm not only interested in comparing SVM's with different values of C, but also in comparing these with another method entirely, e.g. boosted trees, which also has its own set of hyper-parameters that need tuning. Can I evaluate and select these classifiers (algorithms? models?) at the same level?
Caret offers a resampling function that I believe does just this (see vignette). It returns the final models for the hyper-parameter tuning of several classifiers. The final model is the one chosen based on the comparison of the cross-validation performance between for example a SVM with C=1 and C=2, and refit to the entire training set. By comparing the final SVM model with the final boosted tree model, I believe we are selecting on the same level as before. Thus if we select SVM's in favour of boosted trees, it would still be valid to estimate the generalization error on the left out test set (or in the outer CV in the case of a nested CV). Is my reasoning correct?
The elements of statistical learning (Chapter 7) describes model selection as: "estimating the performance of different models in order to choose the best one", but afterwards only describes cross-validation for hyper-parameter tuning (and the necessity of including feature selection and pre-processing in the CV scheme), without mentioning the comparison of different classification methods. In my mind this would also fall under the umbrella of model selection).
This FAQ on the other hand does seem to make a distinction between model selection for hyper-parameters and classifier selection (see scenario 3). That being said, I don't fully understand the reasoning presented there, nor does the final figure seem to be related to the third scenario presented in the text. The claim that this setting automatically leads to a nested CV also does not seem to agree with what I've read about this topic in other places.
In a nutshell, when people talk about model selection using cross-validation, are they referring to tuning a certain model's (e.g. SVM) hyper-parameters and selecting the best SVM model, or can this be generalised to comparing different kinds of models, e.g. classifiers/algorithms (SVM vs boosted trees vs neural nets vs ...).
