In my data I have 35 features and 14 of them are categorical. Half of them have 3 to 4 categories but others have 14 to 28 categories.

One Hot Encoding them would only lead to a sparse matrix with too many features correlated between them.

Do you know how can I handle this problem ?

  • 1
    $\begingroup$ How do you know that there would be "too many" features? Tree based algorithms, and in particular XGBoost can deal with this kind of data pretty well, provided you have enough data. I would suggest making dummies/one-hot encoding, and seeing how well XGBoost performs. You could try using the 21 non-categorical features and the 7 low-cardinality ones, and then add the 7 high-cardinality features and see whether, under cross-validation, your loss gets better or stays the same/gets worse $\endgroup$ – gazza89 Sep 20 '18 at 11:19

There are possibly many ways to tackle this, depending on your data, feature cardinality, etc.:

  • After one-hot-encoding, it may turn out some new features are almost always zero and have negligible statistical significance and you can just drop them
  • Whole features (before encoding) may turn out to be insignificant
  • For some of your categorical features, ordering may actually make sense, like "small,medium,big". In such case, you can just use numerical encoding without increasing number of features
  • You can use binary encoding to reduce dimensionality. There already is an answered question that deals with a somehow similar topic: Binary Encoding vs One hot Encoding

Please refer for this great article for in-depth analysis of different encoding schemes and their performance: https://medium.com/data-design/visiting-categorical-features-and-encoding-in-decision-trees-53400fa65931

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.