8
$\begingroup$

I have a dataset with some variables with MANY categories (one has about 20000, other about 2000, the third about 200). Need to make a multi-class prediction (not binary, but 3 values)

How can I manage them? If I would like to try, for example, a Random Forest, I think max categories are about 50. I suppose creating 200 dummy variables is not an option (2000 or 20000 even worse).

What would be a better idea, to try grouping them somehow, or try another algorithm?

EDIT: May be I need to indicate the NUMBER of observations. My data has about 100000 observations, so, a category with 20000 levels, may be too much (only 5 observations per level). Even 2000 levels (50 observations per level), could be too much?

$\endgroup$
5
  • $\begingroup$ Whether or not this is a good idea can come later, but what keeps you from just including all the categories? $\endgroup$
    – Dave
    Apr 24, 2023 at 11:13
  • 2
    $\begingroup$ Related/duplicate from 2017 with no answers: stats.stackexchange.com/q/300391/36229 $\endgroup$ Apr 24, 2023 at 20:26
  • $\begingroup$ @Dave, may be I'm wrong, but I had the Idea that a Random Forest converts a categorical variable into dummy variables (one-hot-encoding), so the model would face to tune a '22200 categories' dataset. Isn't it too much? $\endgroup$
    – Kaikus
    Apr 25, 2023 at 12:38
  • $\begingroup$ A high-definition image has $1080\times1920\times3\approx6-\text{million}$ pixels for the same number of features, quite a bit more than you have. It might be that you have too many features for your sample size or that there are better ways to model that many categories (e.g., random effects), but it is not a given that $20000$ features is too many. $\endgroup$
    – Dave
    Apr 25, 2023 at 14:51
  • $\begingroup$ @Dave. I have a dataset with about 100000 observations, with a categorical value stored as 'character' with 20000 different values, and another variable with 2000 categories. An 'average' of 5 and 50 observations per category, respectively. This is the dimensionality of my proble. I supposed that, in this case, theese are too many categories, and I should try, or reduce them somehow, or try another algoryth, $\endgroup$
    – Kaikus
    Apr 28, 2023 at 11:23

1 Answer 1

7
$\begingroup$

Until Breiman and the 21st c the historic barrier for working with massively categorical features was computational, for example in ANOVA, inverting a cross-products matrix with too many categories was infeasible.

That said, it's useful to distinguish between massively categorical features vs targets. It's true that random forests (RFs) have trouble modeling targets with more than a few dozen levels. This is not true for massively categorical features.

Breiman's intent with RFs was to redress criticisms of his original 'single iteration' approach to classification and regression trees as being unstable and inaccurate.

What Breiman didn't realize was that any multivariate modeling engine could be plugged into his RF framework, e.g., ANOVA, multiple regression, logistic regression, k-means, and so on, to arrive at an approximating, iterative solution.

Breiman did his work in the late 90s on a single CPU when massive data meant a few dozen gigs and a couple of thousand features processed over a couple of thousand iterations of bootstrapped resampling of observations and features. Each iteration built a mini-RF model, the predictions from which were aggregated into an ensemble prediction of the target.

Today there are dozens of workarounds to modeling massively categorical features which extend Breiman's approach to breaking a large model down to many bite-sized, smaller models, sometimes known as divide and conquer algorithms.

A paper by Chen and Xie discusses D&Cs, A Split-and-Conquer Approach for Analysis of Extraordinarily Large Data https://www.jstor.org/stable/24310963

Another good review is McGinnis' Beyond One-Hot: an exploration of categorical variables https://www.kdnuggets.com/2015/12/beyond-one-hot-exploration-categorical-variables.html

Related to this is the suggestion of impact coding, e.g., Zumel's Modeling Trick: Impact Coding of Categorical Variables with Many Levels https://win-vector.com/2012/07/23/modeling-trick-impact-coding-of-categorical-variables-with-many-levels/

A completely different, non-frequentist approach was made in marketing science wrt hierarchical Bayesian modeling: Ainslie and Steenburgh's Massively Categorical Variables: Revealing the Information in Zip Codes. Their model is easily programmed in software such as STAN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=961571

Hope this helps address your query.

Afterthought FWIW...having poked around with a few of the approaches to massive categorical information including one-hot encoding, hierarchical bayes and impact coding, I came to the opinion that impact coding offered the best results in several nonsignificantly better ways: strongest holdout metrics wrt dependence, minimized metrics of dispersion and easiest to code.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the references :) $\endgroup$
    – Kaikus
    Apr 28, 2023 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.