6
$\begingroup$

I am sure most of us have had this problem at some point.

Take for example the random forest algorithm and its implementation in R. This algorithm can not handle a categorical variable with more than (approximately) 55 categories. The work-around for this problem usually involves manually selecting categories with low counts and combining them into a single category. However, users usually have to decide themselves what should be considered as a "low count" as well as the desired count of the combined category. Different selections of these thresholds can result in different models with different performances.

I was wondering if there was a more mathematical approach to this problem. For example - is there any statistical algorithm that might try to randomly decide to combine different categories and decide these thresholds, and then decide which threshold combination to select based on some metric (e.g. best performance (e.g. F-Score, AUC) of the model based on these combination/thresholds)?

Or is it better to just use some algorithm in the background such as the Genetic Algorithm or Simulated Annealing?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Is there a reason clustering algorithms wouldn't help here? $\endgroup$
    – mkt
    Aug 6, 2022 at 7:15
  • 2
    $\begingroup$ I don't know about "most of us". I would never expect an automated procedure to work well with so many categories, but that's zero help to anyone trying to do this. $\endgroup$
    – Nick Cox
    Aug 6, 2022 at 8:52

1 Answer 1

Reset to default
12
$\begingroup$

The most rational and elegant solution, and best performing in terms of mean squared error of estimates, is to use a method that borrows information across groups: either penalized maximum likelihood estimation, mixed effects model, or a Bayesian model that connects the groups through the use of random effects. But a quick and not-too-dirty solution is to estimate how many parameters the sample size will allow you to estimate and to pool categories with the lowest marginal frequencies until you only estimate that many parameters.

A drastically different approach is to reduce dimensionality by scoring the 55 groups. For example, Fisher's optimum scoring algorithm replaces groups with the mean value of a surrogate variable computed on just that group. Or use subject matter knowledge. For example if you had 55 zip codes you could replace the categories with the median family income that exists within its zip zode, or code as distance from some center point.

Improve this answer
$\endgroup$
6
  • $\begingroup$ @ Frank Harrell: Thank you so much for your answer! $\endgroup$
    – stats_noob
    Aug 6, 2022 at 18:36
  • $\begingroup$ 2) estimate how many parameters the sample size will allow you to estimate and to pool categories with the lowest marginal frequencies until you only estimate that many parameters. $\endgroup$
    – stats_noob
    Aug 6, 2022 at 18:37
  • $\begingroup$ 3) Fisher's optimum scoring algorithm replaces groups with the mean value of a surrogate variable computed on just that group. ..isnt fishers scoring algorithm a variant of the Newton-Raphson optimization algorithm that is used for estimating the parameters of a likelihood function? How is it estimating a surrogate variable across groups? $\endgroup$
    – stats_noob
    Aug 6, 2022 at 18:38
  • $\begingroup$ 1) a method that borrows information across groups: either penalized maximum likelihood estimation, mixed effects model, or a Bayesian model that connects the groups through the use of random effects. $\endgroup$
    – stats_noob
    Aug 6, 2022 at 18:42
  • 1
    $\begingroup$ Sample size papers by Richard Riley (first author) are the best available papers. Simpler versions of that are in RMS book and course notes at hbiostat.org/rms. Without penalization if you wanted to accurately estimate the outcome for every level of the grouping variable you'd need n=96 observations per level if Y were binary. But keep in mind that richer modeling with random effects allows a more continuous way of thinking. $\endgroup$
    – Frank Harrell
    Aug 7, 2022 at 11:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.