Flag redundant categorical variables in a big dataset I have a dataset with ~150 categorical variables and ~150k rows. It is expected beforehand that a number of the categorical variables will be either identical, or nearly so. I would like to code something in R to flag these variables.
My first effort in this regard has been to grab the first categorical variable, then sequentially run Chi-square tests on this variable with the remaining variables in the dataset. This however is producing lots of "false positives". 
For example, I have a date variable, which is yielding sub 0.01 p-values when tested against another variable Y . Now it is totally plausible that the distribution of Y varied with date, so perhaps the Chi-Sq test is inappropriate for this task. I'm really after some means of testing if these categorical variables are almost entirely equivalent, say 95% of rows are equivalent when encoded. 
Any advice?
 A: I've found a solution that works reasonably well. Rather than testing with Chi Square, to see if two variables are similar, I instead take a sample of the data, split it into "train" and "test" subsets, and then fit a Support Vector Machine or a Naive Bayes classifier, using one variable as an outcome and one as a regressor. I then use the classifier to predict on the "test" dataset and see how well the predicted values match the actuals. If the match is sufficiently high, I consider these two variables to be potentially redundant. I've put this test into an algorithm, using a while loop that grabs a reference variable (dependent) from the full list of categoricals, and a for loop that fits these test models on the remaining variables, testing for redundancy. At each step of the while loop, I remove any variables which were predicted strongly from consideration in the next step, and save the flagged redundant variables. 
SVM performs a little better with my data, Naive Bayes however takes only 10% as long to run. 
