3
$\begingroup$

The Wikipedia article on cross validation states

[...] cross-validation can be misused [...] By allowing some of the training data to also be included in the test set – this can happen due to "twinning" in the data set, whereby some exactly identical or nearly identical samples are present in the data set. Note that to some extent twinning always takes place even in perfectly independent training and validation samples. This is because some of the training sample observations will have nearly identical values of predictors as validation sample observations. And some of these will correlate with a target at better than chance levels in the same direction in both training and validation when they are actually driven by confounded predictors with poor external validity.

While I do not completely understand this paragraph (e.g. what is "better than chance levels"?), it makes sense to me that samples with "similar" input values should be all in the test set or all in the training set when doing cross-validation. Are there any published results on (1) how the groups for $k$-fold cross-validation should be chosen, and (2) how the cross-validated estimate may get worse when $k$ increases?

$\endgroup$

1 Answer 1

4
$\begingroup$

Don't have time to write a full answer right now, but here's a start.

Assuming you read German, you can find more readable summaries in my thesis (C. Beleites: Raman-spektroskopische Diagnostik von primären Hirntumoren mit Hilfe weicher chemometrischer Klassifikationsmethoden, 2015, FSU Jena), see the "Validierung" section in the Introduction and part II "Entwicklungen und Untersuchungen zur Validierung von chemometrischen Modellen im Rahmen der Dissertation".

Feel free to contact me if you have questions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.