Using test data sets in simulation study

Question

I'd like to know the correct way to simulate test data for a simulation study. For simplicity, suppose that I want to test a linear regression model. Chapter 7 of ESL explains that the average test error is equal to the expected prediction error, where the average is over the training data.

$\text{Err}_{\mathcal{T}}$ is the test error for a specific training set $\mathcal{T}$. They have an example stating that $\text{Err}_{\mathcal{T}}$ was calculated for 100 simulated training sets. $\sum_{\mathcal{T}=1}^{100}\text{Err}_{\mathcal{T}}/100$ is then an estimate of the expected prediction error $\text{Err} = E(\text{Err}_{\mathcal{T}})$.

Page 220 of ELS:

My question is: must 100 test sets be simulated for the calculation or is each $\text{Err}_{\mathcal{T}}$ calculated using the same test set?

Here is some R code to demonstrate:

library(MASS)
n = 50
m = 200
b0 = 0.5
b = c(1,2,0,0)
rho = 0.5
sigma = 1
iter = 100
p = length(b)
r = matrix(rho, p, p); diag(r) = 1

# test set option A
# x.new = mvrnorm(m, rep(0,p), r)
# y.new = b0 + x.new %*% b + rnorm(m, 0, sigma)

err.t = rep(0, iter)
for (i in 1:iter) {

  # training set
  x = mvrnorm(n, rep(0,p), r)
  y = b0 + x %*% b + rnorm(n, 0, sigma)

  # test set option B
  x.new = mvrnorm(m, rep(0,p), r)
  y.new = b0 + x.new %*% b + rnorm(m, 0, sigma)

  mod = lm(y ~ x)
  pred = predict(mod, x.new, type="response")
  err.t[i] = sum((y.new - pred)^2)/m
}
err = mean(err.t)

Should the test set be generated outside the loop as in option A, or inside the loop as in option B?

The same question goes for a validation set. Suppose I was fitting a LASSO model and want to use a separate data set for model selection, i.e. choose the best tuning parameter, should the validation set be outside or inside the loop?

Answer 1

It seems like in this case they use the same test set $(X, Y)$ in each of the experiments (option A). Such an error estimate can be biased towards this specific test set, i.e. the actual error might be either lower or higher depending on the "difficulty" of the sample. It is clear that if we happen to sample mostly trivial examples for the test set, then our estimate is going to be overoptimistic. This could lead to overfitting when we use this estimate to tune some parameters of the model, e.g. we would choose too simple model.

The alternative value computed for different test samples (option B) would also be a valid estimate of actual model performance, and should be more robust since it utilize more data. I would prefer the option B if I were able to simulate the data. However, in practice, we often deal with finite datasets, which are more difficult to gather. In such case it is much easier to just divide them into train, validation and test set. I think that the authors of the book were simply referring to this paradigm.

Answer 2

I would point out that although @pkubik may be correct in saying it is option A, this is not necessarily the case and I don't think it is possible to say for sure from the information provided in the book.

In the image below I have attempted to recreate the simulation using option B instead. As you can see, it is possible to obtain results very similar to those in the original image using this approach.

The code that generated this image is available here.