I'm having trouble with a basic machine learning methodology question. I understand the concept of not using the same data to both train and evaluate a classifier, and furthermore when there are parameters in an algorithm to be optimized, you should use an independent third test set to get the final reportable performance figures (e.g. recall rate). However, using a single test set to measure performance seems to be problematic because the measures of performance will likely differ depending on how the data is partitioned into training (plus validation) and test sets, especially for small datasets. It would be better to average the results of N different partitions.
For the training stage, this is exactly why people normally use k-fold cross-validation, but if you were to repeat the final test stage with different test sets (i.e. k-fold cross validation), then you are mixing data used for parameter tuning with the test data. This is what all the machine learning references I've seen say not to do. But how to get a good performance measure that is not dependent on partitioning of the data set?
Related to cbeleites's comment:
I'm not sure whether or not I understand the concept of nested cross validation. Here is a check of my understanding. Please comment if I am not correct. Assume I have 1200 labeled samples, and I want to compare 3 different models (e.g. a decision tree algorithm with three different parameter settings). For each model, I would have two cross-validation loops--an outer loop that performs testing and an inner loop that generates the model and also tests it. For the "outer" loop, if I were to choose 4-fold I first make a 75/25 split to create a training/validation set (900 samples) and test set (300 samples). With the 75% portion, I perform k-fold cross validation (in my example below I choose 3-fold, in order to keep the figure simple, so there are 300 samples per fold) and calculate the average performance of the k-folds. I would then re-train the same model (i.e. same parameter settings) using the full 75% portion and then test with the remaining 25%.
I would then repeat this procedure three times using different folds from the outer loop (see figure). The final part is where I'm a bit unsure what is the right procedure. Do I average the four tests from each model (i.e. the folds from the outer loop). What actually is the purpose then of the inner loop? Or am I not doing the nesting correctly at all?