How to handle too many categorical features with too many categories for XGBoost?

Question

In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories.

One Hot Encoding them would only lead to a sparse matrix with too many features correlated between them.

Do you know how can I handle this problem ?

How do you know that there would be "too many" features? Tree based algorithms, and in particular XGBoost can deal with this kind of data pretty well, provided you have enough data. I would suggest making dummies/one-hot encoding, and seeing how well XGBoost performs. You could try using the 21 non-categorical features and the 7 low-cardinality ones, and then add the 7 high-cardinality features and see whether, under cross-validation, your loss gets better or stays the same/gets worse — gazza89, Commented Sep 20, 2018 at 11:19

Marcin · Accepted Answer · 2018-09-20 10:50:03Z

There are possibly many ways to tackle this, depending on your data, feature cardinality, etc.:

After one-hot-encoding, it may turn out some new features are almost always zero and have negligible statistical significance and you can just drop them
Whole features (before encoding) may turn out to be insignificant
For some of your categorical features, ordering may actually make sense, like "small,medium,big". In such case, you can just use numerical encoding without increasing number of features
You can use binary encoding to reduce dimensionality. There already is an answered question that deals with a somehow similar topic: Binary Encoding vs One hot Encoding

Please refer for this great article for in-depth analysis of different encoding schemes and their performance: https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931

1 Answer 1