5
$\begingroup$

I am sure most of us have had this problem at some point.

Take for example the random forest algorithm and its implementation in R. This algorithm can not handle a categorical variable with more than (approximately) 55 categories. The work-around for this problem usually involves manually selecting categories with low counts and combining them into a single category. However, users usually have to decide themselves what should be considered as a "low count" as well as the desired count of the combined category. Different selections of these thresholds can result in different models with different performances.

I was wondering if there was a more mathematical approach to this problem. For example - is there any statistical algorithm that might try to randomly decide to combine different categories and decide these thresholds, and then decide which threshold combination to select based on some metric (e.g. best performance (e.g. F-Score, AUC) of the model based on these combination/thresholds)?

Or is it better to just use some algorithm in the background such as the Genetic Algorithm or Simulated Annealing?

$\endgroup$
2
  • 1
    $\begingroup$ Is there a reason clustering algorithms wouldn't help here? $\endgroup$
    – mkt
    2 days ago
  • 2
    $\begingroup$ I don't know about "most of us". I would never expect an automated procedure to work well with so many categories, but that's zero help to anyone trying to do this. $\endgroup$
    – Nick Cox
    2 days ago

1 Answer 1

10
$\begingroup$

The most rational and elegant solution, and best performing in terms of mean squared error of estimates, is to use a method that borrows information across groups: either penalized maximum likelihood estimation, mixed effects model, or a Bayesian model that connects the groups through the use of random effects. But a quick and not-too-dirty solution is to estimate how many parameters the sample size will allow you to estimate and to pool categories with the lowest marginal frequencies until you only estimate that many parameters.

A drastically different approach is to reduce dimensionality by scoring the 55 groups. For example, Fisher's optimum scoring algorithm replaces groups with the mean value of a surrogate variable computed on just that group. Or use subject matter knowledge. For example if you had 55 zip codes you could replace the categories with the median family income that exists within its zip zode, or code as distance from some center point.

$\endgroup$
6
  • $\begingroup$ @ Frank Harrell: Thank you so much for your answer! $\endgroup$
    – stats_noob
    2 days ago
  • $\begingroup$ 2) estimate how many parameters the sample size will allow you to estimate and to pool categories with the lowest marginal frequencies until you only estimate that many parameters. $\endgroup$
    – stats_noob
    2 days ago
  • $\begingroup$ 3) Fisher's optimum scoring algorithm replaces groups with the mean value of a surrogate variable computed on just that group. ..isnt fishers scoring algorithm a variant of the Newton-Raphson optimization algorithm that is used for estimating the parameters of a likelihood function? How is it estimating a surrogate variable across groups? $\endgroup$
    – stats_noob
    2 days ago
  • $\begingroup$ 1) a method that borrows information across groups: either penalized maximum likelihood estimation, mixed effects model, or a Bayesian model that connects the groups through the use of random effects. $\endgroup$
    – stats_noob
    2 days ago
  • $\begingroup$ Are there any references for 1) and 2)? How would you find out how many parameters the sample size will allow you to estimate? Thank you so much! $\endgroup$
    – stats_noob
    2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.