Remove duplicates from training set for classification

Question

Let us say I have a bunch of rows for a classification problem:

$$X_1, ... X_N, Y$$

Where $X_1, ..., X_N$ are the features/predictors and $Y$ is the class the row’s feature combination belongs to.

Many feature combination and their classes are repeated in the dataset, which I am using to fit a classifier. I am just wondering if it is acceptable to remove duplicates (I basically perform a group by X1 ... XN Y in SQL)? Thanks.

PS:

This is for a binary presence only dataset where the class priors are quite skewed

Answer 1

No, it is not acceptable. The repetitions are what provide the weight of the evidence.

If you remove your duplicates, a four-leaf clover is as significant as a regular, three-leaf clover, since each will occur once, whereas in real life there is a four-leaf clover for every 10,000 regular clovers.

Even if your priors are "quite skewed", as you say, the purpose of the training set is to accumulate real-life experience, which you will not achieve if you lose the frequency information.