Why not use large ($n - [n^{0.75}]$) validation sets in machine learning training loop?

Question

I am seeking feedback in the form of answers to a bold question. Please limit comments to requests for clarification of the question only.

There appears to be an influence on machine learning practice from the theory of cross-validation. However, since machine learning training loops do not actually perform cross-validation, but only a single validation to calculate a gradient, why do most machine learning practitioners use a smaller validation data set than the size of their training data for each epoch?

Why do we, as a community of practicing statisticians, not advocate instead for the use of Jun Shao's $n^{\frac{3}{4}}$ rule when choosing the machine learning training set size? This would guarantee that a machine learning validation set is never smaller than a training set.

I implore you to look at the logo of this web site and ponder how deeply the traditions of $k$-fold (in particular, 5-fold) cross-validation have influenced the world and our own community. In particular, please observe how the logo seems to reinforce the practice of validating on 1/5 of the data after training on 4/5 (when using 5-fold CV).

I propose a better method: train on $n_c \equiv n^{\frac{3}{4}}$, validate on $n_v \equiv n - n_c$ data points. And do not use cross-validation; use a better approximation of Bayesian inference, such as sequential model-based optimization, instead.

If you are training on synthetic data, and/or your data set is so large that you cannot possibly use anywhere close to all of it for training/validation, I suggest training on 128 data points, then validating on 512. This reverses the usual practice of training on 4/5 and validate on 1/5 when applied to a data set of size 640, and has the advantage that each data set is a whole power of two (actually Shao would have suggested training on 127, validating on 513 but this is a big enough step in the right direction that I am willing to compromise, because the $\frac{3}{4}$ exponent was probably not the only one he could have chosen).

Answer 1

Your post seems to be starting from the assumption that we are all making a silly mistake by not using $n^{3/4}$-sized training sets. This is not true, as there are precise senses in which AIC and LOO are the "right thing to do" despite the fact they are not model-selection consistent.

Nothing helps clarify these things better than a simple simulation study. Let's look at a linear model with $N = 625$ samples with the model $$ Y_i = \sum_{j=1}^3 X_j + \epsilon_i $$ with a total of $5$ predictors, all $X_{ij} \sim N(0,1)$. We will consider five models, each containing $(X_{i1}, \ldots, X_{ik})$ for $k = 1,\ldots, 5$. We will do the following:

• Select the best model according to AIC.
• Select the best model according to a train/validation split which takes a training set size of $125$.
• Compute the mean prediction error on a third untouched dataset with $N = 10,000$, using the selected models fit to both training and validation (since the $n^{3/4}$ approach will obviously not do well otherwise).

We will repeat this some number of times and see what we conclude.

set.seed(11235813)
N_rep <- 500

aic_results <- numeric(N_rep)
split_results <- numeric(N_rep)

N <- 625
N_test <- 10000
sigma <- 4

for(k in 1:N_rep) {
  X <- matrix(rnorm(N * 5), nrow = N)
  Y <- rowSums(X[,1:3]) + sigma * rnorm(N)
  
  X_ho <- matrix(rnorm(N_test * 5), nrow = N_test)
  Y_ho <- rowSums(X_ho[,1:3]) + sigma * rnorm(N_test)
  real_test <- data.frame(X = X_ho)
  
  my_data <- data.frame(X = X, Y = Y)
  
  idx_train <- 1:125
  train_set <- my_data[idx_train,]
  test_set <- my_data[-idx_train,]
  
  fit_1 <- lm(Y ~ X.1, data = train_set)
  fit_2 <- lm(Y ~ X.1 + X.2, data = train_set)
  fit_3 <- lm(Y ~ X.1 + X.2 + X.3, data = train_set)
  fit_4 <- lm(Y ~ X.1 + X.2 + X.3 + X.4, data = train_set)
  fit_5 <- lm(Y ~ X.1 + X.2 + X.3 + X.4 + X.5, data = train_set)
  
  split_models <- list(fit_1, fit_2, fit_3, fit_4, fit_5)
  
  eval_model <- function(fit) {
    muhat <- predict(fit, test_set)
    mean(abs(test_set$Y - muhat)^2)
  }
  
  fit_12 <- lm(Y ~ X.1, data = my_data)
  fit_22 <- lm(Y ~ X.1 + X.2, data = my_data)
  fit_32 <- lm(Y ~ X.1 + X.2 + X.3, data = my_data)
  fit_42 <- lm(Y ~ X.1 + X.2 + X.3 + X.4, data = my_data)
  fit_52 <- lm(Y ~ X.1 + X.2 + X.3 + X.4 + X.5, data = my_data)
  
  full_models <- list(fit_12, fit_22, fit_32, fit_42, fit_52)
  
  best_aic <- which.min(Vectorize(AIC)(full_models))
  best_split <- which.min(Vectorize(eval_model)(split_models))
  
  # print(c(best_aic, best_split))
  
  aic_error <- mean(abs(Y_ho - predict(full_models[[best_aic]], real_test))^2)
  split_error <- mean(abs(Y_ho - predict(full_models[[best_split]], real_test))^2)
  
  aic_results[k] <- aic_error
  split_results[k] <- split_error
}

plot(aic_results, split_results)
t.test(aic_results - split_results)

Results AIC outperforms the approach suggested by OP in terms of average prediction error, by a small-but-detectable amount. The reason is that, while AIC will often select models which are too large and take a small hit in accuracy, the $n^{3/4}$ approach will occasionally leave a predictor out. When a predictor gets left out, you take a huge hit in prediction, and its enough to offset the times you selected the correct model when AIC did not.

None of this contradicts any theory in any papers. It is indeed true that the $n^{3/4}$ model selects the correct model more often than AIC. But that is not much consolation if your goal is prediction, which is the case in most machine learning applications.