I have several points which appear duplicates in the feature matrix (same values for the features). These points may have different values of the target variable. What is the appropriate way to handle this case during train test split? Should I just ignore the problem and go on with random split or remove duplicated points ?


It depends on what does a duplicate mean in your data.

If your features contain identification variables (for example, transaction number, student ID, etc), then duplicates are just copies of a same sample. In this case, if their target values ($y$) are the same, you should remove the duplicates. If they have different $y$'s, then this sample is problematic, and you should probably consider removing them altogether.

If your features does not contain identification variables, then it is totally possible that samples with the same feature values are different samples, and therefore possible to take different $y$'s. In this case, you don't need to do anything special.

  • $\begingroup$ Good suggestions for dealing with original data. It should be noted that in bootstrapping (often a better way to proceed than with the separate training and test sets proposed by the OP) you will necessarily end up with many "duplicate" cases in each bootstrap re-sample, duplicates that are not to be removed. $\endgroup$ – EdM Dec 26 '17 at 19:29
  • $\begingroup$ In my case duplicates come from the fact we only partially observe the data: (for example another feature or an id which would help differentiating the rows has been omitted) in such a case I was wondering if removing duplicates may speed up training ... $\endgroup$ – user511005 Dec 26 '17 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.