0
$\begingroup$

Since version 1.5, XGBoost supports categorical data out of the box, which is a convenient way to skip the one-hot pre-processing step and allow for if X in values splits.

For example, given a category animal with values: dog, cat, lizard, snake, XGBoost can now split based on e.g. if animal in {dog, cat}, i.e. treating animal as a single feature, rather than using 4 "sub-features" from the one-hot encoding of animal and doing if X < threshold splits. This works for datasets like:

row, animal
0,   dog
1,   cat
2,   lizard

However, would it be possible to extend this to features with multiple values in the same category? That is, for datasets where the feature of a training instance is a subset of potential categories (multi-hot encoding)? e.g.:

row, animal
0,   {dog}
1,   {dog, cat}
2,   {cat, snake, lizard}

My intuition says no, because essentially each subset becomes a new "value", and the category now is the powerset, which becomes impractical for large cardinality. But I was wondering if there are additional tricks here.

$\endgroup$

1 Answer 1

1
$\begingroup$

Unless you want "cat/lizard/snake" as an extra category (probably not attractive because of many possible combinations, but perhaps an option in some cases), the standard category handling does not work for a multi-membership problem. (One-hot- or dummy-)encoding each individual category is the obvious alternative, as you note.

Alternatives include forms of embeddings that you average. E.g. one option is multi-membership target encoding. One form of this would be obtained by fitting a suitable random effects regression model with a multi-membership random effect (of course, you'd still want to do things like data splitting to avoid target leakage), but much simpler options like just calculating a target encoding for each category ignoring membership in the others and then averaging is also possible (but may not handle categories that differ substantially in prevalence all that well).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.