When doing cross-validation for model selection, I found there are many ways to quote the "standard deviation" for the cross-validation scores (here "score" means an evaluation metric e.g. accuracy, AUC, loss, etc.)
- One way is to calculate the standard deviation on the mean of the scores of $K$ folds (= standard deviation of $K$ folds / $\sqrt K $).
- The second way is to calculate just the standard deviation of the scores of $K$ folds. An example can be found here.
- Another way which I don't quite understand. It seems to calculate the standard deviation of $K$ folds / $\sqrt N$ where $N$ is the size of the dataset...
Personally I think 1) is correct, as we care more about the standard error on the sample mean (here sample mean = the average score of $K$ folds validation) rather than the standard deviation of the sample. Can anyone explain which way is preferred?