I have a dataset of about 3 million observations with around 1000 duplicate cases/rows (found simply by using the duplicated() function). I'm trying to figure out why these cases might have been duplicated.

I'd like to quickly check for systematic differences between duplicated and non-duplicated cases across all variables. However, the data contain a mix of continuous and categorical variables. If I knew ahead of time which variables were continuous and which were categorical, I could do something like Chi-square tests for the categorical variables and T-tests for the continuous variables. But there's hundreds of variables and manually differentiating between categorical vs continuous would probably take too long.

Any suggestions for either a one-size-fits-all-variables approach for this, or a way to logically differentiate between categorical vs continuous and then apply the appropriate tests? The data come straight from a csv so categorical variables haven't already been factorized. Also, categorical variables use numbers as codes (think numerical codes for State, region, etc), not strings, so that won't work either.

Thank you!

  • $\begingroup$ "Continuous" and "categorical" are modeling decisions and, sometimes, are characterizations of levels of measurement. As such these are not properties that can be deduced from any amount of data. That makes it difficult to understand what you are trying to accomplish: what is your objective in conducting this analysis of patterns of duplication? $\endgroup$ – whuber Sep 13 '17 at 13:45
  • $\begingroup$ Good point. I was using "continuous" and "categorical" in the measurement sense, where the former would include variables whose values represent actual numbers (quantities), and the latter's values do not (i.e. nominal variables). Sorry for the confusion. My objective is simply to identify variables on which duplicates and non-duplicates seem to differ in order to try and understand why duplication may have occurred. It's investigative, basically. So for example if duplicates are more likely to originate from some areas than non-duplicates, this would be a useful clue. $\endgroup$ – lost Sep 14 '17 at 2:10
  • $\begingroup$ I realize there's nuance here (e.g. variables can be measured in ordinal categories, which aren't truly continuous, etc), and if I was doing a more formal analysis I would look very closely at each variable and how it was measured. I was just wondering if there is a quick mining-style approach for a situation like this, to help me narrow down what variables might help explain the pattern of duplication, after which I can look more closely at the smaller set of variables. $\endgroup$ – lost Sep 14 '17 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.