Encoding categorical variables that do not have rank/order for classification tasks?

Question

I have primarily worked with classification problems that have numerical features size, height, weight, distance, times, bag of words/tf-idf, so on. However, I am starting to consider working with more datasets that has categorical features. Examples of such features are race, genre (book or movie), color, shape, city name, school, etc. If I had a categorical variable called length that had the levels small, medium, large, it makes sense to recode that as 1, 2, 3 since there is already some hierarchy or order present. However, I am confused about encoding variables that do not have a natural translation to a numerical scale or already have an intrinsic sense of order.

With something like color, imagine the options are red, green, and blue. It does not make sense for red to be 1, green to be 2, and blue to be 3, because that means that red is technically "closer" to green than blue and what does it mean in this case for color to be ranked or have distance? Similarly, imagine we have a simplified list of genres (romance, comedy, drama, scifi, action). If I encode that as romance - 1, comedy - 2, drama - 3, scifi - 4, action - 5, that means in this representation romance is "closer" or more similar to comedy than drama, which really does not make sense. I feel by doing this, we're artificially creating structure and meaning that is not already present.

If you think that encoding in this manner is totally acceptable, I would love to be proved wrong (it would be much more convenient and simple to work with) -- please just explain why. If my interpretation is correct, can you explain how I can address this? What are examples of things that you have done (or commonly accepted practices)? I realize that binary variables are fairly straightforward with the examples above, but not necessarily if I have significantly more options. Note: I primarily work with Python, Pandas, and Sklearn for ML. If you suggest other tools that can help with this, let me know.

Thank you!

Answer 1

One-hot encoding is the most widely used general purpose method for unordered categorical features. It's a non-lossy encoding of the feature set & allows models to handle interactions between categories; furthermore, certain model fitting procedures are amenable to sparse representations of the feature set that one-hot produces, allowing for efficient models using one-hot encoding.

That said, there are alternative encodings that may be useful, particularly in scenarios where dimensionality & data size are an issue:

• Impact coding - map categorical features to a single continuous value representing the conditional probability of the response. This comes with several difficulties including high bias estimates in sparse subspaces of the feature set, as well as potential for data leakage (user must be disciplined in their approach to cross-validation)
• A somewhat related approach is to fit a model on a subset of the one-hot encoded data and extract a smaller set of features (perhaps including interactions between different sets of features). An example of this is RuleFit, which fits tree ensembles to both continuous & categorical features, extracting a concise set of binary features from the fitted trees representing subsets of features (both continuous & categorical) as well as interactions between features.
• For more structured or domain specific data, previously developed embeddings may exist (e.g. word2vec for text data)

As a final note, ordered categorical data will offer a richer set of feature engineering techniques- be mindful of this property of the data.