# Use of variance of estimators in cross-validation

Let's suppose we are using K-fold cross validation on a set of data of dimension $$N_{data}$$. We do not want to fix any parameter but just to get a confidence of the predictive power using the different train-test validation splits provided by the cross validation procedure.

At one iteration we fit that $$K$$ models, each of which uses $$N_{fit}=\frac{K-1}{K}N_{data}$$ for fitting and $$N_{val}=\frac{1}{K}N_{data}$$ for testing. We can call $$Q_i,i=1..K$$ the performance/scores of the fitted model on that testing set.

It is common than to take $$m=\frac{1}{K}\sum_{i=1}^K Q_i$$ as the predicted performance of the model. In scikit-learn then the standard deviation of the $$Q_i$$ is used to get an estimate of the confidence of this estimator (see Ex. 3.1.1.):

https://scikit-learn.org/stable/modules/cross_validation.html

My questions are:

• Is there a reference that shows what his the statistics behind this procedure ?

• If the $$Q_i$$ were indipendent, the standard deviation of the mean of the estimators is:

$$Err= \sqrt{\frac{1}{K(K-1)}\sum_i (Q_i-m)^2}$$,

which is the standard deviation divided by $$\frac{1}{\sqrt{K}}$$. I know that the $$Q_i$$ are not independent (are they?) but doesn't look an overkill to take just the standard deviation of the $$Q_i$$ as a confidence estimator ? (If K is large than the standard deviation does not go to zero for example)