enter image description here

$$f_{i}=A(D/ D_{i})$$ $$e_{j}=L(f_{i}, z^{(j)})$$

It seems to me, that above definition of k-folded cross validation algorithm (from Deep Learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville, 2016) is inconsistent with the common definition of cross - validation. In above algorithm $e$ vector is the vector of loss function calculated for every particular example in the $D$ dataset, and then mean of vector $e$ is the estimation of generalization error. Whereas in standard definition of cross - validation, we calculate test error for each fold and then calculate average of them.

  • 1
    $\begingroup$ What is your question? $\endgroup$ Feb 10 '17 at 19:37
  • $\begingroup$ What is the difference between those two definitions of cross validation $\endgroup$
    – mokebe
    Feb 10 '17 at 19:39

There's really no distinction between those definitions. (Perspective, perhaps?) As the pictured text reads, emphasis mine:

The algorithm returns the vector of errors $\epsilon$ for each example in $\mathbb{D}$, whose mean is the estimated generalization error.

To square this with the definition you give above, note that the mean of all errors is equal to the weighted (by $\frac{|\mathbb{D_i}|}{N}$) average of means over each of k folds. (K-fold cross validation error is a weighted average of errors of particular folds.)

  • $\begingroup$ "note that the mean of all errors is equal to the weighted (by |Di||Di|) average of means over each of k folds.". Could you provide derivation? $\endgroup$
    – mokebe
    Feb 10 '17 at 20:11
  • $\begingroup$ Sorry, I goofed that formula twice—should now follow from expressing each group mean as $\Sigma_{\mathbb{D}_i}\epsilon_j$. $\endgroup$ Feb 10 '17 at 20:44
  • 1
    $\begingroup$ My misconception was a result of assumption that cross validation error is a non - weighted average of errors of particular folds, wheareas cross validation is a weighted average, which is often not highlighted in textbooks. $\endgroup$
    – mokebe
    Feb 10 '17 at 21:38
  • $\begingroup$ Often times texts will sort of hand-wave over the assumption that each of the k folds are of equal size (making the two equivalent). Will jot up an edit shortly. $\endgroup$ Feb 10 '17 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.