1
$\begingroup$

I use cross-validation, with random partitioning, to test the generalization power of a certain classifier (a short-sentences classifier).

The problem is, my dataset contains many almost-identical samples, such as "I do not accept your offer", "I can not accept your offer", etc. So, when I partition the dataset randomly, there are many samples in the train partition that are very similar to samples in the test partition, and I get exceptionally high performance, which is actually over-fitting.

Is there a better way to test the generalization power of my classifier?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.