0
$\begingroup$

Let's suppose we are using K-fold cross validation on a set of data of dimension $N_{data}$. We do not want to fix any parameter but just to get a confidence of the predictive power using the different train-test validation splits provided by the cross validation procedure.

At one iteration we fit that $K$ models, each of which uses $N_{fit}=\frac{K-1}{K}N_{data}$ for fitting and $N_{val}=\frac{1}{K}N_{data}$ for testing. We can call $Q_i,i=1..K$ the performance/scores of the fitted model on that testing set.

It is common than to take $m=\frac{1}{K}\sum_{i=1}^K Q_i$ as the predicted performance of the model. In scikit-learn then the standard deviation of the $Q_i$ is used to get an estimate of the confidence of this estimator (see Ex. 3.1.1.):

https://scikit-learn.org/stable/modules/cross_validation.html

My questions are:

  • Is there a reference that shows what his the statistics behind this procedure ?

  • If the $Q_i$ were indipendent, the standard deviation of the mean of the estimators is:

    $Err= \sqrt{\frac{1}{K(K-1)}\sum_i (Q_i-m)^2}$,

    which is the standard deviation divided by $\frac{1}{\sqrt{K}}$. I know that the $Q_i$ are not independent (are they?) but doesn't look an overkill to take just the standard deviation of the $Q_i$ as a confidence estimator ? (If K is large than the standard deviation does not go to zero for example)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.