1
$\begingroup$

I have binary medical data (6000,200) and I removed all duplicates (even though all records generated are of unique data of different patients- they showed similar pattern and thus duplicate data came in dataset). Then I applied chi square test and discarded non-significant features. Then features reduced to 20 from 200 features...

So now I again checked for duplicates and found only 100 unique samples in the data set (that means only 100 patients data i have now). So now data is of shape (100,20) Is it okay to remove such duplicates even though they belong to different people prior to perform machie learning? But in real world data, duplicates are there... So I feel like I should keep them... May I know any literature also there to support this?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Removal of "insignificant" features is highly problematic. There is no chance you are finding the "right" features, and this will create overfitting and distortion of statistical inference. $\endgroup$
    – Frank Harrell
    Commented Jul 23, 2021 at 12:16

1 Answer 1

Reset to default
3
$\begingroup$

I think you shouldn't even delete duplicate entries before feature selection because they're not real dups and data belongs to different patients. You're manually changing the data distribution by doing so. So, let alone after the feature selection, you shouldn't do it even before it.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.