How to properly report results from Cross Validation, specifically standard deviation

Question

This seems to be a standard question but unfortunately, I have not found any good explanatory resources.

So, my situation is as follows. I would like to evaluate a machine learning model and report its accuracy on unseen data. Since my dataset is really small, I employ cross-validation.

Once I have obtained accuracy values on all folds, I would like to report the mean and the standard deviation of these.

However, several sources on the internet do not directly report the standard deviation. Instead, they tend to divide the standard deviations by the square root of the number of folds first.

I have not found any reasoning on why this is done.

Answer 1

Let accuracy (or any other performance metric) of fold $i$ be $X_i$, out of $n$ folds. You'd likely want to report the overall success of your model considering all the folds, which is usually chosen as the average of them, i.e. the mean, $\bar X$. Assuming $X_i$ are iid, $$\mathbb E[\bar X]=\mathbb E[X_i], \ \ \ \ \ \ \operatorname{std}(\bar X)=\frac{\operatorname{std(X_i)}}{\sqrt{n}}$$

$\mathbb E[X_i]$ and $\operatorname{std(X_i)}$ can be estimated from what you have from each fold, i.e.

$$\forall i \ \ \ \mathbb E[X_i]=\mathbb E[\bar X] \approx \frac{1}{n}\sum_{j=1}^n x_j$$ where $x_i$ are realizations of $X_j$, i.e. in your case accuracy of fold $j$. Similarly, $\operatorname{std}(X_i)$ is estimated from the standard deviations of mean accuracies.