Preprocess categorical variables with many values [duplicate]

Question

I have a dataset that consists of only categorical variables and a target variable. I want to predict the (binary) target variable with the categorical variables.

I am trying to do this in Python and sklearn.

The categorical variables have many different values. I was thinking to create dummy variables for each value in all the categorical variables. However, in the end this results in a sparse data set with thousands of variables.

How would you go about to reduce the number of dummy variables? Would you use Chi2 to select useful features?

Or would you not make dummy variables at all?

Answer 1

There are multiple questions here, and some of them are asked & answered earlier. First, the question about computation taking a long time. There are multiple methods to deal with that, see https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix and the paper by Maechler and Bates.

But it might well be that the problem is with modeling, I am not so sure that the usual methods of treating categorical predictor variables really give sufficient guidance when having categorical variables with very many levels, see this site for the tag [many-categories]. There are certainly many ways one could try, one could be (if this is a good idea for your example I cannot know, you didn't tell us your specific application) a kind of hierarchical categorical variable(s), that is, inspired by the system used in biological classification, see https://en.wikipedia.org/wiki/Taxonomy_(biology). There an individual (plant or animal) is classified first to Domain, then Kingdom, Phylum, Class, Order, Family, Genus and finally Species. So for each level in the classification you could create a factor variable. If your levels, are, say, products sold in a supermarket, you could create a hierarchical classification starting with [foodstuff, kitchenware, other], then foodstuff could be classified as [meat, fish, vegetables, cereals, ...] and so on. Just a possibility.

Orthogonal to the last idea, you could try fused lasso, see Principled way of collapsing categorical variables with many categories which could be seen as a way of collapsing the levels into larger groups, entirely based on the data, not a prior organization of the levels as implied by my proposal of a hierarchical organization of the levels.

Answer 2

Think about the following problem. You have a huge matrix (with let say 1000 rows and 1000 columns). In each cell of this matrix you have one or no values. You need to create a predictive model that predicts the value in a cell given by the row ID and column ID.

The described problem faces the same problem as you do: As input you have only categorical variables (row ID and column ID are categorical) and each categorical variable has many possible values (number of rows and number of columns).

How does this problem is solved? One standard way to solve this problem is a matrix factorization . You basically assign different numerical vectors to each row and each column and then you calculate the value in the cell by applying a function to the vectors corresponding to the selected row and column. For example, in the case of the Non-Negative Matrix Factorization this function is just scalar product of the row-vector and column-vector.

So, if you want to apply the same approach to your problem, you need to map each value of each categorical variable into a numerical vector. And then you use these vectors as inputs to your model-function and as output you get your predictions.

The exact mapping from categorical variables to vectors and / or shape of the function are decided by the model-training.

Another way to approach your problem is inspired by collaborative filtering. To predict a value for a given row and column you need to find similar rows and columns and get values from them. Basically, in your cases it translates to a sort of k-NN (nearest neighbor) approach. Use values of the categorical variables to find row with similar values of categorical variables. Then take the values of the targets from the "neighbors" and combine them (for example by averaging them out, maybe with the weights proportional to similarity measure).