0
$\begingroup$

I am confident that the below methodology creates a bias in that it indirectly uses the test set as part of the training procedure, however, I am having trouble explaining this to my colleagues. I would greatly appreciate any relevant terminology or reading material to help me articulate the problem.

Here is an example scenario:

  • I have a dataset split into three chunks A, B, and C.

  • I use two chunks to train my model, and one to test.

  • I try all permutations of test/train splits (ie using each AB, AC, BC as training), and get different results each time.

  • I report the performance from the best of the splits as the performance of my model on this data.

In reality, this value is an over-optimistic evaluation of my model's performance because I have essentially looked at all the data and chosen the best way to split it. This is cheating because I have looked at my test data before evaluating. The actual performance of my model would be the average performance of all split permutations.

What is the bias I have introduced called?

What is this (common) mistake called?

Where/how can I find a more complete description of this problem?

$\endgroup$
1
  • 1
    $\begingroup$ As you said, you should report the mean over all train/test splits, in which case you basically end up with a form of (stratified) cross-validation. If the performance varies a lot, I would even report individual measures. The approach you describe is a form of cherry picking, and is bad practice. $\endgroup$ Aug 26, 2016 at 0:02

1 Answer 1

1
$\begingroup$

To evaluate your forecast accuracy, what you want to do is cross validation. What you don't want to do is data dredging, data fishing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.