3
$\begingroup$

I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and there are missing values, and I have limited time to play with categorical encodings, I plan to use gradient boosting which behaves nicely with such data.

  1. One-hot encoding will lead to an extremely wide dataset with 10K-length vectors. I've read that this might hurt the performance of my model not only in terms of time taken to train but also in terms of one-hot columns adding much noise preventing the model to concentrate on other features.

  2. Even if I use LightGBM or CatBoost that can handle categorical data without encodings, there is still a possible issue. For each category, there are many values that are rare with some having only one entry in my whole training data. I suppose that taking them into account can lead to model learning ungeneralizable patterns.

On Kaggle forum I've seen an idea to drop the values having less than some number of repetitions. Is it considered a good practice and what are the heuristics for choosing the threshold below which values are dropped?

In the same forum topic there was an idea to first add counts for each value (i.e. if [a, a, b, c, b, b] is an original categorical feature 'x', then new feature 'x_counts' = [2, 2, 3, 1, 3, 3] is added) and then convert rare categorical values to one common 'other' value. It sounds like a better idea since we not only drop the particular value but also preserve the information that it was an extremely rare one.

Any ideas will be highly appreciated.

Improve this question
$\endgroup$
3
  • $\begingroup$ I'd create a RareWord category. Infrequently used proper nouns -- like names -- are quite sensibly added to a RareWord category. In a posterior predictive exercise, a simulated sentence would give you something like "We asked Google to go on line and do a Bob search for a good restaurant." -- which is a perfectly good sentence without higher-order n-grams or context-sensitive assignments of rare words. That is, you may want to find digrams of rarer words that occur together, like "Google" and "search", in which case "Bob search" could be suppressed. $\endgroup$
    – Peter Leopold
    Commented Jun 8, 2019 at 16:07
  • $\begingroup$ By RareWord you mean some new binary feature that thresholds the counts of each categorical value? How would you recommend to choose this threshold if I understood you correctly? Thanks. $\endgroup$
    – Igor
    Commented Jun 8, 2019 at 16:17
  • $\begingroup$ The threshold would be a meta parameter, so you would choose it by cross validation. That's the easy answer. In fact, that is how you would defend your choice. You would make your choice by looking at the exigencies of the rest of the calculation. Would threshold=1 be enough to stabilize the rest? Or 2? I would be quite greedy about the selection at first. Of course you would like it to be as small as possible, but first everything else has to work, so start higher rather than lower. 5? 10? $\endgroup$
    – Peter Leopold
    Commented Jun 8, 2019 at 17:43

1 Answer 1

Reset to default
2
$\begingroup$

You can drop the uncommon words, but this would lead to losing information. There's a number of better approaches:

  • As mentioned in comments, you can group the uncommon words in the "other" category.
  • You can use feature hashing, where multiple words would share same encodings.
  • Another approach is to assign unique labels to a subset of most popular words, and for all the other words you would split them to n-grams and encode the n-grams. This is done by some deep learning algorithms, and probably makes sense for those that look at sequences, rather than bag-of-words.

Notice that the above approaches work also for words that were not present in the training set, but appear on prediction time.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.