I use cross-validation, with random partitioning, to test the generalization power of a certain classifier (a short-sentences classifier).

The problem is, my dataset contains many almost-identical samples, such as "I do not accept your offer", "I can not accept your offer", etc. So, when I partition the dataset randomly, there are many samples in the train partition that are very similar to samples in the test partition, and I get exceptionally high performance, which is actually over-fitting.

Is there a better way to test the generalization power of my classifier?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.