Let's suppose we are using K-fold cross validation on a set of data of dimension $N_{data}$. We do not want to fix any parameter but just to get a confidence of the predictive power using the different train-test validation splits provided by the cross validation procedure.

At one iteration we fit that $K$ models, each of which uses $N_{fit}=\frac{K-1}{K}N_{data}$ for fitting and $N_{val}=\frac{1}{K}N_{data}$ for testing. We can call $Q_i,i=1..K$ the performance/scores of the fitted model on that testing set.

It is common than to take $m=\frac{1}{K}\sum_{i=1}^K Q_i$ as the predicted performance of the model. In scikit-learn then the standard deviation of the $Q_i$ is used to get an estimate of the confidence of this estimator (see Ex. 3.1.1.):


My questions are:

  • Is there a reference that shows what his the statistics behind this procedure ?

  • If the $Q_i$ were indipendent, the standard deviation of the mean of the estimators is:

    $Err= \sqrt{\frac{1}{K(K-1)}\sum_i (Q_i-m)^2}$,

    which is the standard deviation divided by $\frac{1}{\sqrt{K}}$. I know that the $Q_i$ are not independent (are they?) but doesn't look an overkill to take just the standard deviation of the $Q_i$ as a confidence estimator ? (If K is large than the standard deviation does not go to zero for example)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.