Do you need a separate test set when performing nested cross-validation?

Question

I understand that the inner cross-validation error can be biased and overly optimistic so that's why people use nested cross-validation to get unbiased generalization error. But we also have a hold out test set when doing cross-validation, so we can get unbiased generalization error from the hold out test set and the overly optimistic cross-validation error won't be a problem to us.

So from my point of view, the two methods described below are both good. I don't see a point of using nested cross validation unless the dataset is small:

1. Split the dataset into train set and test set, perform cross validation on the train set to select the best algorithm (random forest, logistic regression, etc..) and the set of hyper-parameters that have the lowest cross validation error. Retrain the model on the entire training set, then apply that model on the test set to get the generalization error.
2. We do not split the dataset. Perform nested cross validation on the entire dataset. Do whatever feature selection, hyper-parameter tuning within the inner cross validation. Suppose we run nested cross validation on many different algorithms (random forest, logistic regression, etc..) and different feature selection methods, we finally pick the method that gives us the lowest nested cross validation error (i.e. random forest with filtering feature selection). Then apply that method on the entire dataset to obtain the final model. We don't have a separate test set and the nested cross validation error will be our generalization error.

My questions:

1. Do we still need a hold out test set for nested cross validation? Or is the nested cross validation error a good enough generalization error itself?
2. Is there any advantage of using nested cross validation when we have a large amount of data? (>100000 records) If we have a such big dataset, we will have sufficient samples in the hold out test set. Then why do we need to do nested cross validation when we can have a good estimate of the generalization error on the test set? Especially that the computation cost will be so much higher with bigger dataset.
3. Is my description of how to use nested cross validation correct?(the second method above) Specifically this part: "we finally pick the method that gives us the lowest nested cross validation error (i.e. random forest with filtering feature selection)" Is this the right way to decide untimely what's the best process to train the model on our dataset?

Thank you!

Answer 1

Nested cross-validation is essentially just what you describe in process one but repeated multiple times to avoid bad initial splits of train/test. That is it’s just the cross validated analog of your process 1. Why you’d use nested cross-validation is to reduce the variance in your estimate of generalization error. How much nested cross-validation will reduce variance over the other method depends on a variety of things, namely size of dataset and the homogeneity of the data.

Since it is basically just a cross validated version of process 1, you don’t need a further test set. If the dataset is large enough you’re probably okay with just using process 1 as nested cross validation can be very computationally expensive