Standard Error of Mean Squared Error of different cross validation folds

Question

I would like to calculate the Standard Error (SE) of the Mean Squared Error (MSE) of a test set for a certain model size of nnet. It's purpose is to give the 1 SE to serve in the "one-standard-error-rule" (the program is a bit long, because I need some pre-calculations, however if you scroll down to "the interesting part", the question is quite short) How do I get the SE of the MSE/is my calculation right?:

library(nnet)

# only some prerequisits
set.seed(2718)
OBS <- 1000
x1 <- rnorm(OBS, 10, 4)
x2 <- rnorm(OBS, 20, 8)
y <- 100 - x2*40 + x1 * 10 + x1*(x2**2) * 0.5 + rnorm(OBS, 0, 10)
d <- data.frame(y, x1, x2)
FOLDS <- 5
group_index <- rep(0:FOLDS, length.out = OBS)

# scale
my_scale <- function(x) {
  res <- (x - min(x)) / (max(x) - min(x))
  res
}
d <- data.frame(lapply(d, my_scale))

# nnet/train
nnet_mod <- nnet(y ~ ., data = d[group_index == 0, ], size = 5)

# INTERESTING PART
# ----------------
MSE <- lapply((1:FOLDS), function(i) {
  mean((d$y[group_index == i] - predict(nnet_mod, d[group_index == i, ])) ** 2)
})
names(MSE) <- paste0("Fold", 1:FOLDS)
print("individual MSE's:")
print(MSE)
print("Is the SE of the MSE's equal to the standard dev of the MSE's?")
print(sd(unlist(MSE)))
print("If the current model with size = 5 had the lowest MSE, ..")
print(".. the best model would the smallest net that has MSE smaller than:")
print(mean(unlist(MSE)) + sd(unlist(MSE)))

Answer 1

• No, standard error of the mean is not the same as standard deviation of the observations.
• The "minimum plus 1 standard error rule" described in the Elements of Statistical Learning section 7.10.1 uses the standard error of MSE estimates across the folds. This is the standard deviation as you calculate it divided by $\sqrt{\text{FOLDS}}$.
• These rules may be seen as useful heuristic/rule-of-thumb, and depending on the setup of your selection you may want to adjust the threshold, e.g. by setting the limit to some multiple of the standard error.
• It is important to state clearly that you are using the standard error of the fold MSEs, as it would also make sense to formulate the decision threshold on the basis of the pooled predictions over all cross validation folds. However, they should be similar in size, so unproblematic (as long as the description clearly says what was done) as anyways we're just looking at a pragmatic rule-of-thumb.

---

Some more thoughts about your code:

• Your folds are not independent as you scale once for the whole data set outside the cross validation loop.
  To obtain independent folds, the scaling offset and factor should be calculated in each fold for the respective training split and applied to training and test splits.
  The effect of this incorrect preprocessing is probably not large for your MWE, but I've seen a similar incorrect splitting for PCA pre-processing of highly multivariate data leading to errors being underestimated by an order of magnitude.

• if you look at fold-wise MSEs, by using sapply instead of lapply you immediately get a vector instead of a list, which later on saves you all those unlists.