3
$\begingroup$

When doing cross-validation for model selection, I found there are many ways to quote the "standard deviation" for the cross-validation scores (here "score" means an evaluation metric e.g. accuracy, AUC, loss, etc.)

  1. One way is to calculate the standard deviation on the mean of the scores of $K$ folds (= standard deviation of $K$ folds / $\sqrt K $).
  2. The second way is to calculate just the standard deviation of the scores of $K$ folds. An example can be found here.
  3. Another way which I don't quite understand. It seems to calculate the standard deviation of $K$ folds / $\sqrt N$ where $N$ is the size of the dataset...

Personally I think 1) is correct, as we care more about the standard error on the sample mean (here sample mean = the average score of $K$ folds validation) rather than the standard deviation of the sample. Can anyone explain which way is preferred?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.