0
$\begingroup$

Let’s assume that samples in a dataset are characterized by ID, timestamp, features, and target and each sample is a real observation. After dropping totally-duplicated rows, how should duplicates (with regards to all features except for ID and timestamp that are excluded from the model) be treated while building a predictive model (classification)?

These are the different possibilities I thought of:

  1. Drop all duplicates
  2. Drop all duplicates and assign some way a weight proportional to number of times a unique occurrence was present in the dataset
  3. Keep duplicates
  4. Keep duplicates and exclude them some way from validation/test set(s)

In my opinion, 3. is wrong because duplicates could introduce a bias in the model, 4. could solve the issue but it is not feasible in a (nested) cross validation scenario. Looks like dropping duplicates is the right thing to do: 1. is the standard approach with academic or competition datasets, 2. is an improvement of the latter and i think is the best option but I shouldn’t know how to implement it, also because I am using scikit-learn and its importance weighting in CV is broken.

Which is the correct solution? Are there any other valid approaches (maybe under/over sampling related) for treating duplicates?

Improve this question
$\endgroup$
15
  • 4
    $\begingroup$ Why should duplicates introduce a bias in the model? If you have observed a particular predictor-outcome combination 20 times rather than just once, that gives you valuable information. I would absolutely go with 3. What issue do you see here and want to address? Also, I sincerely hope that dropping all duplicates is not taught in courses, because it simply does not make sense. $\endgroup$
    – Stephan Kolassa
    Jan 16 at 10:53
  • 3
    $\begingroup$ What is "not fair" about having multiple records with the same predictor-outcome combination in the training set? (Of course, I am not talking about erroneous duplications of observations.) If you have 20 male smokers, all of which develop cancer, and 2 male smokers who do not develop cancer, would you really discard "duplicates" until you are left with one male smoker with cancer and one male smoker without cancer? ... $\endgroup$
    – Stephan Kolassa
    Jan 16 at 12:08
  • 3
    $\begingroup$ ... Also, it is no problem at all to have the same predictor-outcome combination in both the training and the test set. To the contrary, if this combination occurs often in your population, then artificially removing such instances from the test set will indeed bias your model, how should it then learn that there is indeed a strong association between the predictor values and this particular outcome? In the example above, if you have male smokers with cancer in the training set, then would you propose explicitly excluding male smokers with cancer from the test set? $\endgroup$
    – Stephan Kolassa
    Jan 16 at 12:10
  • 3
    $\begingroup$ Finally, can you point us to a course or textbook where something like this is advocated? I am sure there are such "courses" on the internet, because there is a lot of misinformation especially around statistics and data science on the web. But a claim like removing all (non-erroneous!) duplicates is really just great evidence that this source is unreliable. (I work in retail forecasting. If all I have is the day of week and the sales value, does that mean that having observed two Tuesdays with zero sales is a problem, and we should remove one of them?) $\endgroup$
    – Stephan Kolassa
    Jan 16 at 12:13
  • 2
    $\begingroup$ @AngelMarcos The first tutorial link says "This could be due to things like data entry errors or data collection methods. ", i.e. they're talking about erroneous duplicates. You have to make sure your duplicates are erroneous, you can't just remove an observation only because it's similar to another one. You have to understand why they are similar before choosing to keep or to drop them (e.g. is it coming from the same person answering twice the same survey? Or is it just because two different persons happened to answer the same way?). $\endgroup$
    – J-J-J
    Jan 16 at 14:29

1 Answer 1

Reset to default
5
$\begingroup$

Assuming your dataset should be a random sample drawn from your population, it depends on what you define as a duplicate. I see two scenarios:

  1. "Duplicate" defined as two (or more) distinct observations that happen to have the exact same features/values.

In this case, you should keep them, as it's just how your population of interest is. You immediately see why if you have a dataset of let's say 2,000 observations, but only 2 features with 2 levels ("gender: man or woman" and "Exam results: pass or fail"). If you drop all the observations that are identical, you'll end up at best with 4 observations: 1 man who passed the exam, 1 man who failed, 1 woman who passed, and 1 woman who failed. So in this case, you would just be deleting useful information, and you'll be unable to infer anything about your population of interest.

  1. "Duplicate" defined as the same observation erroneously recorded twice or more

On the other hand, if you define a duplicate as the same observation incorrectly recorded multiple times, ideally you should drop them, because at some point something has been interfering with the sample random draw. In particular, it may be indicative of something going wrong with the data collection process, like a software bug, some typing error, or even fraud from respondents or interviewers, see Kuriakose, N., & Robbins, M. (2016). Don't get duped: Fraud through duplication in public opinion surveys. Statistical Journal of the IAOS, 32(3), 283-291.

This kind of duplicates may seriously bias estimates, see Sarracino, F., & Mikucka, M. (2017, April). Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation. In Survey Research Methods (Vol. 11, No. 1, pp. 17-44). So it may be a good idea to not ignore them if you know they shouldn't be there.

In some situations, it might not be obvious if duplicates (or near duplicates) fall under the first or second category, in particular when you've not been involved in the study design and data collection stages. If you're not sure, you should investigate the matter further, for example by asking additional information to the people who collected the data (hoping that they won't be trying to cover their tracks if it's a case of fraud from their part).

As a side note, you can find various references to prevent or detect problems related to survey fraud or falsification (e.g. Schwanhäuser, S., Sakshaug, J. W., & Kosyakova, Y. (2022). How to catch a falsifier: Comparison of statistical detection methods for interviewer falsification. Public opinion quarterly, 86(1), 51-81.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.