I am using cross-validation to estimate the prediction error of my model. I am using a metric M to measure this prediction error.
Using 10-fold CV, I obtain the value of the metric M for each fold. (Please ignore SD_M for now).
M SD_M result.1 707.4018 196.3860 result.2 1094.0445 260.6073 result.3 821.9250 181.8182 result.4 656.3086 128.1662 result.5 1096.4073 256.0398 result.6 843.6550 192.0989 result.7 588.9200 136.4374 result.8 928.6556 197.5693 result.9 735.6646 159.7934 result.10 792.4319 194.4807
From here, I want to estimate the generalization error, that is to say I want a point estimate of the value of my metric M for arbitrary new data.
I reasonably choose the mean of M across the folds as point estimate of my generalization error.
My question is: What is the standard error SE of this point estimate? Or a confidence interval for this point estimate?
I have several choices and I don't know which one (or if either) is appropriate: I am using R notation but hopefully this is clear for everyone.
SE = sd( c(707.4018, 1094.0445, ..., 792.4319) )
In this case, this yields
SE = sd( c(707.4018, 1094.0445, ..., 792.4319) ) / sqrt(10)since I am estimating the standard error of the mean.
SD_Mcolumn is the standard error of M for each fold. Since M is the MSE,
SD_Mis obtained by using the formula for the standard error of the mean of the squared residuals of this fold:
SD_M = sd(fold_squared_residuals)/sqrt(fold_size).
SE = mean(SD_M) is also a condidate! It is worth about
I am quite confused, I think I'm mixing up concepts. What are the meanings of these 3 values?
EDIT: I edited (clarified) the question quite heavily so the current answers do not really address it. Feel free to have a go!