If I'm splitting my data in training, validation and test data to assess different (sub)sets of features for my task.
What are the consequences if I (by mistake) split my data incorrectly? In the following cases:
- part of the training and validation overlap
- part of the validation and test overlap
- part of the test and training overlap
In case 1, the wrong classifier from the training step could be selected since the classifier trained on the overlapping part of the data has a higher chance of being selected.
In case 2, the test step will rate the classifier better than it is since the classifier was chosen based on part of the test data.
In case 3, the test step will rate the classifier better than it is since the classifier was trained on part of the test data.
Is my reasoning correct? Could I add something to it?