Label encoding for high-cardinality features in boosted decision trees - what to do with unseen labels?

Question

I have a categorical feature with very high-cardinality (on the order of 1000s of unique IDs). RIght now, I am using label encoding along with XGBoost, because from what I understand, decision trees don't require dummy encoding of categorical variables.

Let's say I train my model, and then when generating predictions, my high-cardinality feature contains an unseen label. If the model has been trained on a categorical feature with labels 1 through 100, what happens when I need to predict on something with a label of 101. Ideally I'd like XGBoost to default to the most likely value given the sample it was trained on.

What is the best way to deal with this type of situation?

Answer 1

Stratified cross-validation includes a proportional number of samples from each class in each CV fold. This has its limits, though -- If you have 3 examples of one class and 5 folds, there's no way to both include each sample exactly once and include at least one sample of each class in each fold.

I would examine whether it's feasible to combine (or even exclude) the low-occurrence categories.

Answer 2

If you want to have an unseen category be replaced by the most likely previously seen category, then you could consider the new new category as a missing value and impute the value as a separate preprocessing step. Or have the classification algorithm do the imputation (details on how this is done (or not done) depend on the implementation of the software you're using).

Instead of treating the unseen category as a missing value it is of course possible to encode all previously unseen categories to an "unseen" category (kind of label 101 in the question). But you would need to have some training data having this category also so that things work out as expected. This is because otherwise what happens (or may happen, depends on implementation) is for example if training with data having categories 1-100 produces a decision tree having a split >99 and then this model is applied to new data with category 101 then it would incorrectly be considered having the same effect as category 100...

Depending on the data, one possibility to get such training data is to take all very rare categories (e.g. those seen only in one data point as they may have very limited value anyway) and put those to a "rare/unseen" category. If there are no such rare categories in the data then it could be possible to take a (small) sample of data points from the training set, change the categories of the given feature to the "unseen" category and then append this fake data to the actual training set and train with that. Though this should be considered a bit risky business due to forging data with upsampling part of the original dataset and class labels should not be biased etc.

In this approach the good thing is that if you combine data with different rare/unknown categories together then the classifier will likely learn to not trust that category value too much and the other features will count in more (one could also see this as a sort of inherent imputation if you wish). Which on the other hand then comes close to what considering the unseen values as missing gives you. But it may be conceptually easier to consider an "unseen" category than handling them as missing values, and of course also to make a distinction between real missing values and unseen values.