5
$\begingroup$

I am using cross-validation to estimate the prediction error of my model. I am using a metric M to measure this prediction error.

Using 10-fold CV, I obtain the value of the metric M for each fold. (Please ignore SD_M for now).

                  M     SD_M
result.1   707.4018 196.3860
result.2  1094.0445 260.6073
result.3   821.9250 181.8182
result.4   656.3086 128.1662
result.5  1096.4073 256.0398
result.6   843.6550 192.0989
result.7   588.9200 136.4374
result.8   928.6556 197.5693
result.9   735.6646 159.7934
result.10  792.4319 194.4807

From here, I want to estimate the generalization error, that is to say I want a point estimate of the value of my metric M for arbitrary new data.

I reasonably choose the mean of M across the folds as point estimate of my generalization error.

My question is: What is the standard error SE of this point estimate? Or a confidence interval for this point estimate?

I have several choices and I don't know which one (or if either) is appropriate: I am using R notation but hopefully this is clear for everyone.

  • SE = sd( c(707.4018, 1094.0445, ..., 792.4319) )

In this case, this yields SE=171.33.

  • SE = sd( c(707.4018, 1094.0445, ..., 792.4319) ) / sqrt(10) since I am estimating the standard error of the mean.

This yields SE=54.17.

  • The SD_M column is the standard error of M for each fold. Since M is the MSE, SD_M is obtained by using the formula for the standard error of the mean of the squared residuals of this fold: SD_M = sd(fold_squared_residuals)/sqrt(fold_size).

SE = mean(SD_M) is also a condidate! It is worth about SE=190.3.

I am quite confused, I think I'm mixing up concepts. What are the meanings of these 3 values?

EDIT: I edited (clarified) the question quite heavily so the current answers do not really address it. Feel free to have a go!

$\endgroup$
  • $\begingroup$ I'm not sure I understand your question, but this paper may be relevant/helpful: Bengio & Grandvalet (2003), No Unbiased Estimator of the Variance of K-Fold Cross-Validation (pdf). $\endgroup$ – Lucas Aug 28 '15 at 18:45
  • $\begingroup$ Indeed the paper is extremely relevant to my question, thank you. About my question, can you point me to what you don't understand? Given the lack of response, I imagine I am not as clear as I thought I was! $\endgroup$ – asac - Reinstate Monica Aug 28 '15 at 21:21
2
$\begingroup$

The second approach is reasonable. If the 10 measurements of $M$ were independent, then this is just the standard error of the mean.

Since you are cross-validating, the datasets used in each round and therefore the estimates of $M$ are not independent. So you are probably going to underestimate the standard error a bit. The paper by Bengio & Grandvalet (2003) suggests that there is not much you can do about this bias.

I ran a toy example where the bias seemed small. I fitted a distribution over 10 states with a histogram. The fit was evaluated using average log-likelihood, $L$. Using 100 data points and leave-one-out cross-validation, I got

$$\hat L = 3.5641, \quad \hat{SEM} = 0.7133.$$

This is an average over 10,000 cross-validations. Since I know the true distribution, I can repeat the cross-validation experiment many times with different datasets. I can also generate independent training sets (of 99 data points, as in leave-one-out cross-validation). Using independent datasets, I got

$$\hat L = 3.5636, \quad \hat{SEM} = 0.7185.$$

This is of course only one simple example (code here), and in general the bias might be larger.

$\endgroup$
1
$\begingroup$

I feel like there is a bit of confusion here, stemming from the relationship between MSE and SD. What concerns me is that the word residuals has not appeared, which makes it hard to understand precisely what you want the standard deviation of?

To be precise, if we have some fixed predictor $f$, and data $\{(x_1, y_1), \dots (x_n, y_n) \}$, our residuals are of the form $r_i = y_i - f(x).$ The sample standard deviation of the residuals is equal to:

$$ \sqrt{\sum_i \frac{r_i^2}{n}} = \sqrt{MSE}. $$

Now note, in this case our predictor is not fixed, in fact our predictor is being sampled from a distribution for each fold. Now consider the following model, given your regression algorithm $A$:

1) A dataset D = $\{(x_1, y_1), \dots (x_n, y_n) \}$ is drawn from a distribution by drawing $x_i \sim P(X)$, $y_i \sim P(Y|x_i)$.

2) One more $(x,y)$ is drawn $x \sim P(x)$, $y \sim P(y|x)$

3) A predictor $f$ is trained using your algorithm, $f = A(D)$.

4) A residual is calculated, $r = y - f(x)$.

If we followed this process, we could establish the sample mean and sample standard deviation of residuals of our algorithm. Unfortunately, in real life data and computational power are limited, so we resort to cross validation. This only approximates this process because the residual samples we draw are not independent.

For example, in five fold CV we train on 80% of the data and test on 20%. These residuals that we get from our 20% were all obtained using the same predictor $f$. Essentially as long as we are using the same data over and over again, the residuals will not be independent. That being said, in practice estimating the SD of the residuals by summing all of your MSE and taking the square root is reasonable and standard.

If you were truly trying to calculate the SD of mean squared error itself please comment on this answer and explain why you are interested in this, and I'll respond.

$\endgroup$
  • $\begingroup$ I have edited my question, I hope it is clearer now. I don't really understand how your answer is related to my question. I do know the definition and purpose of (cross-)validation, generalization error of an algorithm, etc. $\endgroup$ – asac - Reinstate Monica Aug 28 '15 at 22:18
  • $\begingroup$ I'm sorry. I don't mean to frustrate you. To be clear about why I was confused it was the original output, which makes it look like you were getting "standard deviations" of single numbers since it says "SD_M" and there is one number called "M." But you have a process that for each fold produces some estimate of error, and you want to know how much that estimate varies. My question is, why do you want to know how much the error of each fold varies, instead of knowing how the error for each prediction varies? $\endgroup$ – jlimahaverford Aug 28 '15 at 22:28
  • $\begingroup$ You are not frustrating me, I was just stating that I didn't get your answer. My ultimate goal is indeed to know how the error for each prediction varies. But I am wondering why we are not using the additional information of SD_M to evaluate it (see my second question). I realize that my post was confusing, this is why I split it into two questions. Thanks for your help! $\endgroup$ – asac - Reinstate Monica Aug 28 '15 at 22:43
  • $\begingroup$ Actually the end goal is rather to have a confidence interval for the generalization error of my algorithm, where "error" means MSE since it is the metric I have chosen. Since the point estimate for the generalization error has several of source of variation, I am wondering how I can estimate its accuracy. Hence the initial question asking how to get the standard deviation of the cv error. $\endgroup$ – asac - Reinstate Monica Aug 28 '15 at 22:45
  • $\begingroup$ This is why I was saying focus on the atomic samples. You're sampling a distribution of squared residuals. Use all of your squared residuals to estimate the variance of your squared residuals. I was arguing that if you know the variance of the squared residual distribution, $v$, then if you take an average of $n$ squared residuals (MSE) the variance will be $v/n$. $\endgroup$ – jlimahaverford Aug 29 '15 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.