6
$\begingroup$

I would like to make/explore several alternative formulations of random forest (link) and am looking for the current state of the science for evaluating performance.

In the reference file Leo provides "microarray", "DNA", "glass", and "spectral" data to exhibit the performance of his method. Are these available? How do I get them in order to first reproduce the results, and explore performance of changed parameter settings? How general and authoritative are they?

Other references that I have thus far found include (this), (this), and (this). Can you suggest rigorous, modern, and relatively complete list of quality classification benchmarks to be used to compare the performance of very similar random forests?

Improve this question
$\endgroup$
4
  • $\begingroup$ I'm voting to close this question as off-topic because it is too broad and requires opinion. $\endgroup$
    – Michael R. Chernick
    Nov 22, 2017 at 17:32
  • 1
    $\begingroup$ @MichaelChernick - I think Stefan Wager and Stephan Kolassa answered what I was looking for to my satisfaction. This question was asked and answered 4 years ago, and something like 6 other CV members upvoted Stefan's answer, implying they understood both the question and answer and felt the answer was relevant. So what brought you to this question? $\endgroup$
    – EngrStudent
    Nov 22, 2017 at 17:48
  • 1
    $\begingroup$ People can answer questions to your satisfaction but that doesn't automatically make the question on topic for this site. I think the decision as to what is off topic for this site has evolved over the years. $\endgroup$
    – Michael R. Chernick
    Nov 22, 2017 at 17:52
  • 1
    $\begingroup$ Given that it is requesting how to benchmark random forests for performance, you might move it to SO. I like having answers to my previous questions because I revisit them. You can also notice that this has been viewed 618 times. That suggests that there were at least 618 times it might have been relevant to someone looking for an answer to a question. $\endgroup$
    – EngrStudent
    Nov 22, 2017 at 18:10

2 Answers 2

Reset to default
7
+50
$\begingroup$

I think random forests are still mostly used in the form they were introduced by Breiman in his 2001 paper. There have been some attempts to improve them by e.g. moving beyond majority voting (http://link.springer.com/chapter/10.1007/978-3-540-30115-8_34), but my impression that this stuff isn't main-stream practice. You can find a good recent review of random forests in Elements of Statistical Learning (http://www-stat.stanford.edu/~tibs/ElemStatLearn/).

The datasets used by Breiman can be found at http://archive.ics.uci.edu/ml/. These datasets are well known classics. The downside is that they are not very large compared to some other datasets out there. That being said, I think the UCI datasets are a great place to start your investigations.

Finally - I think there's still a lot of good work to be done on random forests; the field is far from complete. Good luck!

Improve this answer
$\endgroup$
2
$\begingroup$

One very relevant paper is Fernández-Delgado, Cernadas, Barro & Amorim, "Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?", JMLR, 2014. The authors evaluated many classifiers, among them multiple versions of Random Forests, on the entire UCI repository as of that time and find that Random Forest variants indeed perform best. It seems like specific variants of Random Forests may work better for specific classes of problems, but overall, plain vanilla Random Forests work very well indeed.

Of course, the UCI repository has grown from the 121 datasets the authors used to (currently) 394 datasets (although probably not all of these are classification), so it might make sense to update that study.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ There are likely many more than 179 classifier tools as well. $\endgroup$
    – EngrStudent
    Nov 22, 2017 at 18:08
  • $\begingroup$ You can use datasets from OpenML. A selection of datasets to use them for benchmarking was done here: arxiv.org/abs/1708.03731 There you also find a description on how to download them. $\endgroup$
    – PhilippPro
    Dec 8, 2017 at 14:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.