When doing cross-validation for model selection, I found there are many ways to quote the "standard deviation" for the cross-validation scores (here "score" means an evaluation metric e.g. accuracy, AUC, loss, etc.)

  1. One way is to calculate the standard deviation on the mean of the scores of $K$ folds (= standard deviation of $K$ folds / $\sqrt K $).
  2. The second way is to calculate just the standard deviation of the scores of $K$ folds. An example can be found here.
  3. Another way which I don't quite understand. It seems to calculate the standard deviation of $K$ folds / $\sqrt N$ where $N$ is the size of the dataset...

Personally I think 1) is correct, as we care more about the standard error on the sample mean (here sample mean = the average score of $K$ folds validation) rather than the standard deviation of the sample. Can anyone explain which way is preferred?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.