10
$\begingroup$

Does anybody know a good book/webpage to start learning the techniques of cross validation?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

This website has great information.

In particular, the fourth section of this PDF is what you're looking for

Improve this answer
$\endgroup$
7
$\begingroup$

If cross-validation is to be used for model/feature selection, it is worth bearing in mind that it is possible to over-fit the cross-validation statistic and end up with a model that performs poorly, and the optimised cross-validation statistic can be a severly optimistic performance estimate. The effects of this can be surprisingly large. See Ambroise and McLachlan for an example of this in a feature selection setting and Cawley and Talbot for an example in a model selection setting.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It is good to mention it, but it should be double said that those are examples of misusing or overtrusting CV, not some drawbacks of the method itself. $\endgroup$
    – user88
    Commented Jun 23, 2011 at 10:46
  • 1
    $\begingroup$ indeed, however it is a way in which it is quite commonly misused - so it is important to be aware of when learning about cross-validation! More cross-validation is often a good solution, i.e. nested cross-validation, or as Stone puts it "double-cross" validation. The problem affects pretty much any feature or model selection criterion that is optimised to get a model; there is nothing special about cross-validation in this sense. $\endgroup$
    – Dikran Marsupial
    Commented Jun 23, 2011 at 12:40
0
$\begingroup$

I would also recommend Cross-Validation by Payam Refaeilzadeh, Lei Tang, and Huan Liu.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.