Is cross-validation better/worse than a third holdout set?

Question

I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different folds of cross-validation. If you do X-fold cross-validation and find that the test accuracy is about the same across folds, is this better justified/more robust than having one training set, one test set, and a third validation (a la Kaggle)?

Answer 1

CV by definition only tells you about the training data. It is completely agnostic to external independent data.

Terminology is important here, if a sample has influenced the model building process then it is a training sample. If something is labelled 'test set' that is not compatible with being in a CV loop, whether an inner coefficient optimising or outer hyperparameter optimising loop (since the inner loop will be affected by iterations of the outer loop).

A validation set is static and is often used to determine so called hyperparameters, any decision about the data that influences its nature prior to model building, or any model adjustment that will guide the calculation of the model.

A test set must be independent, not have included any decision about the data generation process, data processing or model building and must represent how you hope to use your model in the real world. It is agnostic of the model, it only exists to answer the question, 'will this model work in this particular situation? ' a test set tells us if the model generalises sufficiently for the scope of the test set.

If ' every data point is used for training and testing among the different folds of cross-validation' then you are discussing fold-held-out samples, not test set holdout samples. In such cases the terminology is being used wrong.

 If you do X-fold cross-validation and find that the test accuracy is about the same across folds, is this better justified/more robust than having one training set, one test set, and a third validation (a la Kaggle)?

No. It is not better justified nor more robust.

If what you mean is that you have a static test set and you peek at it within every CV loop, this is an abuse of the test set. If you mean that you have an outer CV loop, then you just have a second layer in your training approach. Or maybe you meant 'hold out' accuracy rather than 'test'?

What a consistent accuracy across folds tells us is that for the training set the model is consistent. This could be because we have a super generalisable model or because our training set was too narrowly defined. The importance of a validation and test set is to tease apart which of these possibilities you have.

As with any statistical inference, the test set only applies to populations that have the same properties as the test set, so a new test is needed for new populations or new use cases.

Some further resources:

https://datascience.stackexchange.com/questions/18339/why-use-both-validation-set-and-test-set

https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets

https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7

********EDIT***********

I fell victim to the problem of terminology confusion myself. I have edited this to make it more consistent with the Wiki article as that appears to be the most common usage on this site (apologies for not taking the time to write some ML code to do NLP and create some statistics to try to falsify this hypothesis ;-) ). I come from a science background and this use of 'validation' and 'test' seems the wrong way round to me, but I'll try and align with the general usage.