I have a dataset of about 3 million observations with around 1000 duplicate cases/rows (found simply by using the duplicated() function). I'm trying to figure out why these cases might have been duplicated.
I'd like to quickly check for systematic differences between duplicated and non-duplicated cases across all variables. However, the data contain a mix of continuous and categorical variables. If I knew ahead of time which variables were continuous and which were categorical, I could do something like Chi-square tests for the categorical variables and T-tests for the continuous variables. But there's hundreds of variables and manually differentiating between categorical vs continuous would probably take too long.
Any suggestions for either a one-size-fits-all-variables approach for this, or a way to logically differentiate between categorical vs continuous and then apply the appropriate tests? The data come straight from a csv so categorical variables haven't already been factorized. Also, categorical variables use numbers as codes (think numerical codes for State, region, etc), not strings, so that won't work either.