Convert categorical data with large number of levels to numeric data and what kind of mapping to use

Question

I have a dataset with 20,000 rows and 11 columns. Out of the 11 columns 10 are categorical. Out of the 10, three have very large number of levels.

i.e. levels >60.

One of the variable is basically phone model names(for eg. SonyEricssonK700c ,GT-I9192,etc.), hence a simple alphabetical order mapping may not make sense.

The problem here is random forest then cannot be used for categorical variables with levels greater than 53.

Additionally, I also wanted to try K-means clustering on the data, hence, I would need to convert these variables to numerical.

So far what I thought of doing was using :scale(as.numeric(x),center=TRUE,scale=TRUE) #where x is the column I want to convert to numeric.

Answer 1

Ideally, you should use heuristic rules to reduce the number of levels for the phone model variable. For instance, you could create a new feature called phone manufacturer by parsing the name of each phone model. Also, if you have any additional metadata available for each phone model -- e.g., network speed, CDMA vs. GSM, LTE indicator, etc. then you can create additional features that have smaller number of levels than the original phone model variable.

Converting a categorical feature into numeric is an out-an-out mistake. If your plan is to use a technique (e.g., k-means clustering) that can not handle categorical fields, then here's one approach you can consider: create a binary numeric field (0/1) for every level in your original categorical field. For instance, if your original phone model field has 61 unique levels, then you would create 61 binary fields. One such field could be called GT_I9192, and this field will be equal to 1 for each record that has phone_model = 'GT-I9192', and 0 otherwise.

There are two caveats with this approach though: (1) Some techniques, such as k-means, don't work well with such binary features. Please search on this site for suggestions about which distance measures or alternative techniques should be used in such situation. (2) If you create such binary flags from the original phone model variable (with >60 levels) you will get very sparse data, which again, could create issues. I'd recommend reducing the levels of the original field (using some suggestions I've provided above), before creating the binary flags.