0
$\begingroup$

As an extension to this question, for ML problems where it makes sense to remove duplicates (ie: identical data & target variables) from your distribution, in which scenarios would it (if at all) also make sense to de-duplicate the corresponding feature vectors before learning?

For example: lets say we are classifying images of standalone digits. Since the goal is to learn the shape concept vs usage frequency of numbers, we desire a balanced yet unique dataset for each digit. After feature engineering (be it manual or deep learned), syntactically different images of conceptually the same digit may end up encoded with identical feature vectors. In the spirit of data balancing, since some techniques may randomly duplicate the minority classes while others may fiddle with their vectors to synthesize new data-points, is it wrong then to de-duplicate these vectors as well?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

After feature engineering (be it manual or deep learned), syntactically different images of conceptually the same digit may end up encoded with identical feature vectors. In the spirit of data balancing, since some techniques may randomly duplicate the minority classes while others may fiddle with their vectors to synthesize new data-points, is it wrong then to de-duplicate these vectors as well?

Yes, it's wrong. Consider an extreme case, prediction from a model is also a kind of representation. Would you say that it is ok, to de-duplicate the binary classification data into two samples after making hard predictions because the data got represented into two categories? Obviously not.

Consider an example, say that you have a written digit recognition data, something like MNIST. You gathered samples written by 500 people for each of the digits. Would it be ok to leave only the digits written by a single person "because they conceptually represent same digits"? Obviously not. You want to have the data written by different people because while they all wrote the same digits, you want your algorithm to be able to recognize the small differences in the writing styles. On the opposite, you want to have a lot of samples from each of the categories representing the same thing, to learn the natural variability. Without this data, your algorithm would easily overfit to the training samples and would not be able to deal with real-life data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ perhaps I missed something: I’m talking about having an internal step for de-duplicating training vectors, well after data sampling/cleaning & long before predicting. I agree from a representative data standpoint “you want your algorithm to be able to recognize the small differences in the writing styles” but afterwards if the pipeline already vectorized that natural variability for some instances of the same class into the same vector (deeming those differences negligible), then what’s the harm in de-duplicating? $\endgroup$
    – eliangius
    Commented Sep 11, 2021 at 20:45
  • $\begingroup$ @eliangius perhaps I’m missing something, the representations are exactly the same? Sounds unlikely that different data would lead to exactly the same floating-point vectors. If they are not, they are not duplicates. $\endgroup$
    – Tim
    Commented Sep 11, 2021 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.