I have a dataset with about 100K samples described mostly by categorical features. The number of unique values in the categories range from 20 to almost 7000. Since these are categorical values and there are missing values, and I have limited time to play with categorical encodings, I plan to use gradient boosting which behaves nicely with such data.
One-hot encoding will lead to an extremely wide dataset with 10K-length vectors. I've read that this might hurt the performance of my model not only in terms of time taken to train but also in terms of one-hot columns adding much noise preventing the model to concentrate on other features.
Even if I use LightGBM or CatBoost that can handle categorical data without encodings, there is still a possible issue. For each category, there are many values that are rare with some having only one entry in my whole training data. I suppose that taking them into account can lead to model learning ungeneralizable patterns.
On Kaggle forum I've seen an idea to drop the values having less than some number of repetitions. Is it considered a good practice and what are the heuristics for choosing the threshold below which values are dropped?
In the same forum topic there was an idea to first add counts for each value (i.e. if [a, a, b, c, b, b] is an original categorical feature 'x', then new feature 'x_counts' = [2, 2, 3, 1, 3, 3] is added) and then convert rare categorical values to one common 'other' value. It sounds like a better idea since we not only drop the particular value but also preserve the information that it was an extremely rare one.
Any ideas will be highly appreciated.