1
$\begingroup$

I am aware that there are many answers to many posts on this topic (see: 1,2, 3). Generally, the common answer is that this is 'best practice', it avoids 'data leakage' or overfitting.

Is there a scientific basis and proof to these claims? One academic source mentionned was page 222 of this book, but I cannot see why an absolute measure of fit, like the R2, should be calculated on a final testing set.

Similarly, I looked at this paper which only mentions in the discussion the potential need for a final testing dataset.

Improve this question
$\endgroup$
9
  • $\begingroup$ The answer is always the same, because you do not want your model to use any data from the test set when evaluating a model, since this will inevitably lead to overly optimistic results. That's why you do your k-fold (or something else, repeatedly), and then test at the end on a separate set. $\endgroup$
    – user2974951
    Commented Jan 31 at 13:16
  • 1
    $\begingroup$ I've read that answer but is there a proof that "this will inevitably lead to overly optimistic results"? Why would it "inevitably lead to overly optimistic results", since you computed our measure of fit e.g. R2 on the test data that's part of the k-fold process? $\endgroup$
    – CyG
    Commented Jan 31 at 13:18
  • $\begingroup$ Because the model was trained on the test set data as well, and so it will probably perform better on this set than if it didn't see it at all. There are also many other reasons for why this might happen, testing multiple times (inadvertently or not) on the same set until you find a model which (seemingly) performs better on the test set. For an analogy, consider the case where you have to study for a subject for which you have to take a test in the end, would you perform better if you would see said test before the actual test? $\endgroup$
    – user2974951
    Commented Jan 31 at 13:26
  • $\begingroup$ Of course it depends on the actual method that is cross-validated whether results will indeed be overoptimistic. In principle you could define a method in which information from test data is used in a way that it makes matters worse, and in that case data leakage will not normally result in overoptimism. But we would normally use methods in which such information improves results, and then data leakage means we use better information in CV than we would have in a real situation with new data. $\endgroup$
    – Christian Hennig
    Commented Jan 31 at 13:28
  • 1
    $\begingroup$ I don't think it is necessarily or even usually true that "the model was trained on the test set data as well." The point of a training-set/test-set split is precisely to see how a model trained on one set of data will perform on another. I wouldn't modify such a model to better fit the test set. I think only under special circumstances would an additional, "validation" set be needed. However, I see the value of choosing a final model (not judging its performance) based on all available data. $\endgroup$
    – rolando2
    Commented Jan 31 at 13:40

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.