Using random sampling, is it better for the samples to be disjoint?

Question

Background

I would like to measure the performance of a model trained on 3k samples, because this number of samples might be feasible to obtain in practice. I have a larger set of samples to choose from (about 25k), but generating the model is expensive in terms of time and computational effort, so I can only do about 5 runs.

In each run, I will pull 3k samples for training, and test on the remaining samples.

Question

Should the five sets of 3k samples used for training be disjoint? In other words, will this produce a better estimate of this model's performance than five sets of 3k samples selected independently at random? Note that in neither case is it possible to use all samples for training at least once.

Answer 1

Using disjoint data across runs is preferable. Training sets that overlap will have some positive correlation in the fitted models, so the variance of the average test error will be greater,

Var($\sum_i X_i$) = $\sum_i$ Var($X_i$) + $2\sum_{i<j}$ Cov($X_i$,$X_j$).

If you are restricted to only fitting 5 models I would do 5-fold cross validation; use 20k points to train and 5k points to test (overlapping training data, disjoint test data).

There is a trade-off between training size and number of test error samples. The larger the training size the smaller any given test error variance (variance of $\beta$ in regression behaves like $O(n^{-1/2})$ for instance). However the more test error samples the smaller the variance of their average (variance of mean also $O(n^{-1/2})$).

The choice here should not depend on how large (3k) you expect future test sets to be, but rather how much training data you have and how much data you need to fit each model reasonably well.

If you can afford to do this with disjoint train/test data it is preferable, should you have this luxury.