11
$\begingroup$

I have researched the internet|literature a lot on multiclass prediction to find out what is a realistic limit for the number of classes that can successfully be used for estimation when using a RandomForest method.

The literature body on text mining sometimes comes up with really large numbers of classes (>1000), while most other "classical" cases described have a class count less than 6-8. Most of them describe handmade algorithms specifically designed for the particular problem, though, while I am interested in the performance of standard RF implementations (in R, for example).

I have even started to analyse simulated data to find out more about it, but the problem is to generate data that simulates a lot of multiple classes yet has meaningful and realistic predictors.

I know that the results depend largely on the number of observations in every class and the balance between class outcomes. For my data, I can safely assume that there will be enough observations per class, so that I can balance the data accordingly.

So I am curious whether people have applied standard RandomForest implementations to multiclass problems with a class count >>10. Note that I am not talking about separating the estimation into multiple one-vs-all problems.

Does anybody here have some real-life experience with that kind of data?

Improve this question
$\endgroup$
6
  • $\begingroup$ have a look at random ferns - it is fast and it might deal with more classes than random forests because it combines trees in a different way (cvlab.epfl.ch/files/content/sites/cvlab2/files/publications/…). Anyway, 1000 are a lot. It seems more a recommender system rather than a classifier $\endgroup$
    – Simone
    Apr 22, 2013 at 5:58
  • $\begingroup$ Thanks Simone for the note on Random Ferns. You are right, 1,000 is a lot of classes. In my usecase, I'd like to use something in the order of 100 classes. $\endgroup$
    – dp0377
    Apr 22, 2013 at 9:25
  • $\begingroup$ What implementation would you be using? I'm think the vanilla R implementation has a limit of 33 classes. Not sure about other implementations. $\endgroup$
    – JEquihua
    May 25, 2013 at 13:52
  • $\begingroup$ I do not know if it qualifies as an answer. I have aggressively dug into variable importance sampling and found configurations where the importance signal gets lots at a certain count of classes. This is using the 'randomForest' and 'Boruta' libraries. The model can approximate - it is a universal approximator (albeit one with a constant leaf assumption) so it can represent the data, but in terms of determine what is significant when things are in fact significant, there are limits. $\endgroup$
    – EngrStudent
    Apr 16, 2016 at 3:32
  • 1
    $\begingroup$ @Simone this is a lot of classes and recommender system might be a solution, but there are classification problems where you have thousands of classes, e.g. in NLP models when you predict words; ImageNet dataset commonly used in deep learning papers has also thousands of classes. $\endgroup$
    – Tim ♦
    Dec 14, 2018 at 9:58

1 Answer 1

Reset to default
5
$\begingroup$

I have at least one experience doing so. For the NHTS 2017 dataset, I have modeled a number of variables. Notably, random forests perform quite well on predicting vehicle ownership per household (using most of the other household-level variables as features), somewhat outperforming logit models (which are, for whatever reason, state-of-the-art in travel modeling). There are a dozen classes here.

On the other hand, modeling individuals' work schedules (jointly hour leaving to go to work and hour leaving from work) has a large quantity of combinations. After some data preprocessing, there are over 200 classes. Random forest models perform abysmally here, in terms of accuracy. I get about 20% accuracy for an RF model with optimized max depth, and almost 60% accuracy for a logistic regression. Interestingly, the log loss of the RF model is still lower than that of the logistic model.

These results ended up as an extended abstract at TRB. You can read the paper unpaywalled here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why using classes at all for work schedule prediction? Why not a bivariate joint distribution on the times? $\endgroup$
    – kjetil b halvorsen ♦
    Nov 21, 2021 at 12:58
  • $\begingroup$ @kjetilbhalvorsen What do you mean? $\endgroup$
    – Him
    Nov 21, 2021 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.