Is "don't tune based on test" a small sample problem?

Question

I am trying to wrap my head around some of the principles of Machine Learning, and in particular:

Why separate test and validation sets?

The error rate estimate of the final model on validation data will be biased (smaller than the true error rate) since the validation set is used to select the final model After assessing the final model on the test set, YOU MUST NOT tune the model any further! (What is the difference between test set and validation set?)

For conceptual clarity, I would like to understand whether that rule and train-validate-test tri-partition is motivated by small sample considerations, or whether there is a "more fundamental" reason to keep held-out data for unbiased measurement of model performance (part of what made me wonder is Frank Harrel's answer on the aforementioned question, which I would like to understand better, in particular in as much as it relates to small vs. large sample issues: https://stats.stackexchange.com/q/129125).

To make things more concrete, suppose we have a DGP described by $p(y,\textbf{X})$. Our job is to build a process $\pi$ which, when provided with a random sample $s_n = (y_1,\textbf{X}_1), \dots, (y_n,\textbf{X}_n)$ selects a model $\pi^{s_n}$ which associates any $\textbf{X}_i$ in the domain with a predicted value $\pi^{s_n}(\textbf{X}_i)$ for $y_i$.

Here, I am assuming that for practical reasons (think memory constraint), $n$ is fixed to some finite and "relatively low" value. So when I talk about a large sample, I am not referring to the size of $n$, but rather to the ability to generate, at least in principle, an arbitrarily large number of random samples $s^1_n, s^2_n, s^3_n, \dots$.

For concreteness, consider an imagenet-on-steroid kind of situation where my database would be made of trillions and trillions of images --- arbitrarily close to the universe of all possible human-created images. However, I can only use about 1,000,000 to decide which model to select as part of the $\pi$ procedure, again, maybe because of memory constraints.

What I care about here is the performance of the whole process. That is, on average, how well does the selected model predict new instances from the distribution according to some relevant metric. For example, I might care about $\mathbb{E}_{s^n, (\textbf{X}_i,y_i)}(|\pi^{s_n}(\textbf{X}_i) - y_i|)$.

Then, it seems the average performance over a collection of random samples $s^n$ and sets of observations $(\textbf{X}_i,y_i)$ provides an unbiased estimate of the metric I am interested in.

For example, I might compute average of my metric over 100 random sample $s^{1,000,000}_1,\dots, s^{1,000,000}_{100}$ and 100 collections of 50,000 pairs $(\textbf{X}_i,y_i)$. If I then compare two procedures $\pi_1$ and $\pi_2$ according to those averages, I am free to pick the one with the highest average metric. And my selection being based on the metric does not seem to influence the unbiasedness of the metric itself (where, again, bias is with respect to $\mathbb{E}_{s^n, (\textbf{X}_i,y_i)}(|\pi^{s_n}(\textbf{X}_i) - y_i|)$).

(In fact, variance issues aside, it seems the size of the collections of pairs $(\textbf{X}_i,y_i)$ doesn't even matter for unbiasedness?)

So here, it seems there is no need for a final held-out test-set to provide an unbiased estimate of the performance of the procedure I decided to pick? Or am I missing something?

Sorry for the long and convoluted question, and the somewhat unrealistic (?) memory constraint scenario. I hope it helps clarify the nature of my question about sample sizes, but let me know if my question remains unclear.

Answer 1

Fundamentally, the idea that you'd want an evaluation on unseen data comes from a concern that tuning hyper-parameters on data has the potential to overfit. After all, a sufficiently exhaustive search over a sufficiently large hyper-parameter space could theoretically be equivalent to directly fitting the model to that data. That does not mean that with a sufficiently large test set (or a sufficiently good cross-validation or bootstrapping scheme) you could not get a good estimate of the performance (possibly even a better or more efficient one).

Examples of overfitting that illustrate that a simple train-validation split could be dangerous (and even cross-validation) include:

• Since you mentioned ImageNet, have you seen the "Do ImageNet Classifiers Generalize to ImageNet?" publication, which found that creating a new test set according to basically the same procedure led to 11% - 14% drop in accuracy for ImageNet? ImageNet would certainly be a case where incentives are heavily stacked towards making people take choices on test set performance (and not necessarily care as much about true generalization except that it might make it easier to do better on the test set).
• I've certainly seen cases where with many (millions of) records, cross-validation estimates appeared to be over-optimistic compared to a hold-out set (e.g. here or here).
• There's also the "One Standard Error Rule" for selecting the penalty parameter in e.g. LASSO regression, for which one can do a rather exhaustive hyperparameter search (leading to overfitting to the validation set).

Answer 2

In low signal:noise situations as in clinical or behavioral studies, data-set splitting is risky unless you have on the order of 20,000 cases. As Harrell explains:

This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly.

Your focus on "the whole process"

What I care about here is the performance of the whole process.

allows the following approach if you accept the bootstrap principle. You use the bootstrap to evaluate "the whole process" efficiently without completely setting aside a data subset.

First, use your process to develop the model on the entire data set.

Then repeat "the whole process" on a bootstrapped sample of the data, and see how well it works on the original full data set. Do that repeatedly starting with multiple bootstrapped samples and evaluating the model developed via "the whole process" starting from each bootstrapped sample on the full data set. Make sure that you actually repeat "the whole process" on each bootstrap sample.

Insofar as bootstrap sampling from your data set represents taking your original data set from the underlying population, you have thus evaluated how well "the whole process" might be expected to work on repeated samples from the underlying population. You can use this method to estimate the bias from the modeling process and correct the original model if desired. Perhaps you shouldn't think of this as a validation of the model per se, but it is a validation of the modeling process.

With very large data sets the split-sample approach may well be more efficient. But for those of us who work with modestly sized data sets this bootstrap validation is a more efficient use of the available data. With respect to that frequent "small sample problem," separate train/validation/test sets shouldn't be used at all.