Let's say I have a classification problem with $c$ classes. For this, I have a data set containing $N$ distinct feature vectors with $n$ features. Let's say $N$ is of the order of $10^5$, and both $c$ and $n$ are of the order $10$, so that there is enough training data to make statistical reasonable statements.
I have now three different classifiers (say a RandomForest, a NeuralNetwork and a SVM) which I want to train on the data set and then get an estimate of how well each of the classifiers performed and how well each classifier generalizes. Each classifier has hyper parameters (e.g. tree depth for the RandomForest, number of layers in the NeuralNetwork, C value for SVM etc.).
What is now the best way to decide, which classifier performs best? So how can I say "The best RandomForest has a tree depth of $x$ and performs $p$% better than the best NeuralNetwork (which has $h$ hidden layers)?"
My approach would be the following:
- For each of the three different classifiers, define a parameter grid of the hyperparameters, which should be analyzed.
- For each of the the three classifiers do an individual nested cross validation: According to this question on this site, the inner loop of the nested CV selects from the previously defined parameter grid the best set of hyperparameters. The outer loop then tells me, how stable this choice of hyperparameters is. If the standard deviation between the scores of the $k$ outer resulting models is small, then I know that the choice of hyperparameters is stable and not strongly dependent on the subset of the data I used for training.
- The process in step 2 allows me identify the best hyperparameters for each of the three classifiers. I fix these hyperparameters. Lets assume my model is stable and the hyperparameters do not vary strongly between the folds.
- I use the results of the outer cross validation from step 2, to get an unbiased estimate, how well each of the three classifiers performs. The one with the highest score in the nested CV is the one which presumably performs best on unseen new data.
- If I wanted to use one the three classifiers for further classification of unseen data, I would select the one with the highest nested CV score, as mentioned in step 4, and retrain this classifier with all data I have.
Is this a valid approach? Can I use the results of the nested CV to get an estimate of how well the classifier performs on unseen data or do I have to make a new k-fold CV with the best set of hyperparameters and use the results of this as my estimate?
Additionally: Is it valid to perform the nested CV for the three classifiers independently - as presented here - or do I have to do the following:
Perform nested CV where in the inner loop not only the hyperparameters of one classifier are tuned but each of the three classifiers with their respective hyperparameter grid is accessible. In this approach I would not know, how well the "best" of each of the three classifiers performs, right?