1
$\begingroup$

I am seeking feedback in the form of answers to a bold question. Please limit comments to requests for clarification of the question only.

There appears to be an influence on machine learning practice from the theory of cross-validation. However, since machine learning training loops do not actually perform cross-validation, but only a single validation to calculate a gradient, why do most machine learning practitioners use a smaller validation data set than the size of their training data for each epoch?

Why do we, as a community of practicing statisticians, not advocate instead for the use of Jun Shao's $n^{\frac{3}{4}}$ rule when choosing the machine learning training set size? This would guarantee that a machine learning validation set is never smaller than a training set.

I implore you to look at the logo of this web site and ponder how deeply the traditions of $k$-fold (in particular, 5-fold) cross-validation have influenced the world and our own community. In particular, please observe how the logo seems to reinforce the practice of validating on 1/5 of the data after training on 4/5 (when using 5-fold CV).

I propose a better method: train on $n_c \equiv n^{\frac{3}{4}}$, validate on $n_v \equiv n - n_c$ data points. And do not use cross-validation; use a better approximation of Bayesian inference, such as sequential model-based optimization, instead.

If you are training on synthetic data, and/or your data set is so large that you cannot possibly use anywhere close to all of it for training/validation, I suggest training on 128 data points, then validating on 512. This reverses the usual practice of training on 4/5 and validate on 1/5 when applied to a data set of size 640, and has the advantage that each data set is a whole power of two (actually Shao would have suggested training on 127, validating on 513 but this is a big enough step in the right direction that I am willing to compromise, because the $\frac{3}{4}$ exponent was probably not the only one he could have chosen).

$\endgroup$
7
  • 3
    $\begingroup$ Don't you see a problem with the training set being much smaller than the actual sample size? This way you are heavily favoring less flexible models to more flexible ones. This argument is used for favoring leave-one-out cross validation to K-fold cross validation, but that is peanuts compared to how strongly it applies here. What benefits does your approach have compared to the traditional one? I wonder what assumptions are behind the claimed optimality of Shao's result and how relevant they are to different applied problems. $\endgroup$ Oct 28, 2020 at 20:29
  • 2
    $\begingroup$ We know LOOCV and AIC are asymptotically equivalent, and AIC is an efficient model selection criterion. Thus your approach is inefficient (in a precise technical sense), and it seems it is far from it. I guess all boils down to assumptions, though... $\endgroup$ Oct 28, 2020 at 20:34
  • 2
    $\begingroup$ It is important to be extremely precise in what your objectives are when you advocate for larger test sets. See, for example, this paper. Jun Shao's result does not contradict the minimax optimality of tools like AIC or LOOCV for the task of prediction, as opposed to model selection. $\endgroup$
    – guy
    Oct 28, 2020 at 21:50
  • 1
    $\begingroup$ For another example of the need to be precise, to contrast with the comment of @RichardHardy there is a precise sense in which AIC is not an efficient model selection tool: it is easy to prove it is model selection inconsistent in the very simplest models. But it is minimax optimal in a predictive sense. $\endgroup$
    – guy
    Oct 28, 2020 at 21:53
  • 2
    $\begingroup$ @guy, good points. Also, when you say AIC is not efficient, you are using the word "efficient" in a nontechnical sense which might be misleading. I would use the word "inconsistent" (which you use later on) but not "inefficient", because in the technical sense AIC is efficient. $\endgroup$ Oct 29, 2020 at 6:08

1 Answer 1

2
$\begingroup$

Your post seems to be starting from the assumption that we are all making a silly mistake by not using $n^{3/4}$-sized training sets. This is not true, as there are precise senses in which AIC and LOO are the "right thing to do" despite the fact they are not model-selection consistent.

Nothing helps clarify these things better than a simple simulation study. Let's look at a linear model with $N = 625$ samples with the model $$ Y_i = \sum_{j=1}^3 X_j + \epsilon_i $$ with a total of $5$ predictors, all $X_{ij} \sim N(0,1)$. We will consider five models, each containing $(X_{i1}, \ldots, X_{ik})$ for $k = 1,\ldots, 5$. We will do the following:

  • Select the best model according to AIC.
  • Select the best model according to a train/validation split which takes a training set size of $125$.
  • Compute the mean prediction error on a third untouched dataset with $N = 10,000$, using the selected models fit to both training and validation (since the $n^{3/4}$ approach will obviously not do well otherwise).

We will repeat this some number of times and see what we conclude.

set.seed(11235813)
N_rep <- 500

aic_results <- numeric(N_rep)
split_results <- numeric(N_rep)

N <- 625
N_test <- 10000
sigma <- 4

for(k in 1:N_rep) {
  X <- matrix(rnorm(N * 5), nrow = N)
  Y <- rowSums(X[,1:3]) + sigma * rnorm(N)
  
  X_ho <- matrix(rnorm(N_test * 5), nrow = N_test)
  Y_ho <- rowSums(X_ho[,1:3]) + sigma * rnorm(N_test)
  real_test <- data.frame(X = X_ho)
  
  my_data <- data.frame(X = X, Y = Y)
  
  idx_train <- 1:125
  train_set <- my_data[idx_train,]
  test_set <- my_data[-idx_train,]
  
  fit_1 <- lm(Y ~ X.1, data = train_set)
  fit_2 <- lm(Y ~ X.1 + X.2, data = train_set)
  fit_3 <- lm(Y ~ X.1 + X.2 + X.3, data = train_set)
  fit_4 <- lm(Y ~ X.1 + X.2 + X.3 + X.4, data = train_set)
  fit_5 <- lm(Y ~ X.1 + X.2 + X.3 + X.4 + X.5, data = train_set)
  
  split_models <- list(fit_1, fit_2, fit_3, fit_4, fit_5)
  
  eval_model <- function(fit) {
    muhat <- predict(fit, test_set)
    mean(abs(test_set$Y - muhat)^2)
  }
  
  fit_12 <- lm(Y ~ X.1, data = my_data)
  fit_22 <- lm(Y ~ X.1 + X.2, data = my_data)
  fit_32 <- lm(Y ~ X.1 + X.2 + X.3, data = my_data)
  fit_42 <- lm(Y ~ X.1 + X.2 + X.3 + X.4, data = my_data)
  fit_52 <- lm(Y ~ X.1 + X.2 + X.3 + X.4 + X.5, data = my_data)
  
  full_models <- list(fit_12, fit_22, fit_32, fit_42, fit_52)
  
  best_aic <- which.min(Vectorize(AIC)(full_models))
  best_split <- which.min(Vectorize(eval_model)(split_models))
  
  # print(c(best_aic, best_split))
  
  aic_error <- mean(abs(Y_ho - predict(full_models[[best_aic]], real_test))^2)
  split_error <- mean(abs(Y_ho - predict(full_models[[best_split]], real_test))^2)
  
  aic_results[k] <- aic_error
  split_results[k] <- split_error
}

plot(aic_results, split_results)
t.test(aic_results - split_results)

Results AIC outperforms the approach suggested by OP in terms of average prediction error, by a small-but-detectable amount. The reason is that, while AIC will often select models which are too large and take a small hit in accuracy, the $n^{3/4}$ approach will occasionally leave a predictor out. When a predictor gets left out, you take a huge hit in prediction, and its enough to offset the times you selected the correct model when AIC did not.

None of this contradicts any theory in any papers. It is indeed true that the $n^{3/4}$ model selects the correct model more often than AIC. But that is not much consolation if your goal is prediction, which is the case in most machine learning applications.

$\endgroup$
12
  • $\begingroup$ @user5339 I was looking through your code above and I did not see a training/validation epoch-style setup. It looks like you are fitting a brand new model each time through your N_rep loop; please confirm if that is not the case. Also please be advised that in machine learning, the training/validation loop includes multiple epochs, after each of which the previously trained model undergoes additional training in an attempt to reduce the validation loss (training loss function evaluated on a validation data set). This is the context of my proposal, and cross-validation is not the context. $\endgroup$
    – brethvoice
    Oct 29, 2020 at 14:21
  • $\begingroup$ @brethvoice I am doing a simulation experiment to assess the long-run frequency characteristics of your proposed approach. Within each experiment, I perform model selection first using your splitting strategy with distinct train/validation sets and then checked performance on a large test set. As far as I can tell this is a valid way to assess your claims. Each iteration generates completely new data. $\endgroup$
    – guy
    Oct 29, 2020 at 14:31
  • 1
    $\begingroup$ @brethvoice I’d also like to add that I could have given a mathematical argument for how you are misinterpreting the results of Shao, but based on the tenor of your OP I guessed you would not find it convincing because you have already made up your mind (for example, commenting that people here might not be “open minded” enough to accept your proposal). I had hoped a simulation experiment would be more convincing. $\endgroup$
    – guy
    Oct 29, 2020 at 14:33
  • $\begingroup$ @user5339 please provide a mathematical argument. Thank you for taking the time to run an experiment, even though it is not one which is a valid way to assess my claims relating to machine learning training/validation loops which keep the trained model from the prior epoch. What does the term training epoch mean to you? $\endgroup$
    – brethvoice
    Nov 2, 2020 at 14:51
  • 1
    $\begingroup$ @brethvoice (i) The paper you suggest is not relevant, and Yang's paper has been cited 690 times. (ii) The paper is available here. (iii) If you read Shao's paper, it is very clear that all of his theoretical results are couched in the linear regression setting. It's even in the title: "Linear Model Selection by Cross-validation." One expects that similar results hold for nonlinear models, but the point is that any suggestion that your approach is uniformly better than LOO is not even true in Shao's setting. $\endgroup$
    – guy
    Nov 3, 2020 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.