6
$\begingroup$

Let’s assume that samples in a dataset are characterized by ID, timestamp, features, and target and each sample is a real observation. After dropping totally-duplicated rows, how should duplicates (with regards to all features except for ID and timestamp that are excluded from the model) be treated while building a predictive model (classification)?

These are the different possibilities I thought of:

  1. Drop all duplicates
  2. Drop all duplicates and assign some way a weight proportional to number of times a unique occurrence was present in the dataset
  3. Keep duplicates
  4. Keep duplicates and exclude them some way from validation/test set(s)

In my opinion, 3. is wrong because duplicates could introduce a bias in the model, 4. could solve the issue but it is not feasible in a (nested) cross validation scenario. Looks like dropping duplicates is the right thing to do: 1. is the standard approach with academic or competition datasets, 2. is an improvement of the latter and i think is the best option but I shouldn’t know how to implement it, also because I am using scikit-learn and its importance weighting in CV is broken.

Which is the correct solution? Are there any other valid approaches (maybe under/over sampling related) for treating duplicates?

$\endgroup$
15
  • 8
    $\begingroup$ Why should duplicates introduce a bias in the model? If you have observed a particular predictor-outcome combination 20 times rather than just once, that gives you valuable information. I would absolutely go with 3. What issue do you see here and want to address? Also, I sincerely hope that dropping all duplicates is not taught in courses, because it simply does not make sense. $\endgroup$ Commented Jan 16, 2023 at 10:53
  • 4
    $\begingroup$ What is "not fair" about having multiple records with the same predictor-outcome combination in the training set? (Of course, I am not talking about erroneous duplications of observations.) If you have 20 male smokers, all of which develop cancer, and 2 male smokers who do not develop cancer, would you really discard "duplicates" until you are left with one male smoker with cancer and one male smoker without cancer? ... $\endgroup$ Commented Jan 16, 2023 at 12:08
  • 4
    $\begingroup$ ... Also, it is no problem at all to have the same predictor-outcome combination in both the training and the test set. To the contrary, if this combination occurs often in your population, then artificially removing such instances from the test set will indeed bias your model, how should it then learn that there is indeed a strong association between the predictor values and this particular outcome? In the example above, if you have male smokers with cancer in the training set, then would you propose explicitly excluding male smokers with cancer from the test set? $\endgroup$ Commented Jan 16, 2023 at 12:10
  • 4
    $\begingroup$ Finally, can you point us to a course or textbook where something like this is advocated? I am sure there are such "courses" on the internet, because there is a lot of misinformation especially around statistics and data science on the web. But a claim like removing all (non-erroneous!) duplicates is really just great evidence that this source is unreliable. (I work in retail forecasting. If all I have is the day of week and the sales value, does that mean that having observed two Tuesdays with zero sales is a problem, and we should remove one of them?) $\endgroup$ Commented Jan 16, 2023 at 12:13
  • 4
    $\begingroup$ @AngelMarcos The first tutorial link says "This could be due to things like data entry errors or data collection methods. ", i.e. they're talking about erroneous duplicates. You have to make sure your duplicates are erroneous, you can't just remove an observation only because it's similar to another one. You have to understand why they are similar before choosing to keep or to drop them (e.g. is it coming from the same person answering twice the same survey? Or is it just because two different persons happened to answer the same way?). $\endgroup$
    – J-J-J
    Commented Jan 16, 2023 at 14:29

1 Answer 1

11
$\begingroup$

Assuming your dataset should be a random sample drawn (without replacement) from your population, it depends on what you define as a duplicate. I see two scenarios:

  1. "Duplicate" defined as two (or more) distinct observations that happen to have the exact same features/values.

In this case, you should keep them, as it's just how your population of interest is. You immediately see why if you have a dataset of let's say 2,000 observations, but only 2 features with 2 levels ("gender: man or woman" and "Exam results: pass or fail"). If you drop all the observations that are identical, you'll end up at best with 4 observations: 1 man who passed the exam, 1 man who failed, 1 woman who passed, and 1 woman who failed. So in this case, you would just be deleting useful information, and you'll be unable to infer anything about your population of interest.

  1. "Duplicate" defined as the same observation erroneously recorded twice or more

On the other hand, if you define a duplicate as the same observation incorrectly recorded multiple times, ideally you should drop them, because at some point something has been interfering with the sample random draw. In particular, it may be indicative of something going wrong with the data collection process, like a software bug, network issues, some typing error, or even fraud from respondents or interviewers, see Kuriakose, N., & Robbins, M. (2016). Don't get duped: Fraud through duplication in public opinion surveys. Statistical Journal of the IAOS, 32(3), 283-291.

This kind of duplicates may seriously bias estimates, see Sarracino, F., & Mikucka, M. (2017, April). Bias and efficiency loss in regression estimates due to duplicated observations: a Monte Carlo simulation. In Survey Research Methods (Vol. 11, No. 1, pp. 17-44). So it may be a good idea to not ignore them if you know they shouldn't be there.


In some situations, it might not be obvious if duplicates (or near duplicates) fall under the first or second category, in particular when you've not been involved in the study design and data collection stages. It is also possible that you have these two kinds of duplicates in your dataset.

If you're not sure how to identify what kind of duplicates you have in front of you, you should investigate the matter further, for example by asking additional information to the people who collected the data (hoping that they won't be trying to cover their tracks if it's a case of fraud from their part).

Looking at the details of the raw data –beyond your variables of primary interest– may be useful too to identify duplicates generated by software bugs or other input errors (e.g. if you've been collecting data online, looking at timestamps and IP addresses may be useful for this purpose).

The bottom line here is to look for any piece of secondary information that could help you distinguish between observations that look identical on the surface, including information that is not directly available in your dataset.

As a side note, you can find various references to prevent or detect problems related to survey fraud or falsification (e.g. Schwanhäuser, S., Sakshaug, J. W., & Kosyakova, Y. (2022). How to catch a falsifier: Comparison of statistical detection methods for interviewer falsification. Public opinion quarterly, 86(1), 51-81.)

$\endgroup$
2
  • $\begingroup$ The bottom line here is to use any piece of secondary information that could help you distinguish between observations that look identical on the surface. Wow, what a summary of regression! $\endgroup$
    – Dave
    Commented Apr 16 at 16:33
  • $\begingroup$ @Dave ahahah! :) I edited the answer following your comment, the idea was relative to collecting additional data, not relative to analysis. $\endgroup$
    – J-J-J
    Commented Apr 16 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.