I'm reading a paper that says that the authors used "bagged random forests". I couldn't understand this because as far as I know a random forest is a kind of bagging on its own. So a random forest is a bag of trees. But a bagged random forest?! Would that mean a bag of random forests with each random forest having 10 or 100 trees?!

This is the paper: http://users.cs.fiu.edu/~lzhen001/activities/KDD_USB_key_2010/docs/p243.pdf

enter image description here


enter image description here

  • $\begingroup$ The paper has author's name and also list their email address. May be, you could email them and ask directly otherwise it would be purely speculative. I have never heard a term called "Bagged Random Forest". It might simply mean random forest. $\endgroup$ – forecaster May 7 '14 at 23:56
  • $\begingroup$ @forecaster I followed your suggestion and I wrote to the authors, I hope they will have some time to answer to this dilemma $\endgroup$ – rapaio May 8 '14 at 7:59
  • $\begingroup$ @rapaio have they replied yet? $\endgroup$ – Jack Twain May 18 '14 at 11:21
  • $\begingroup$ @AlexTwain No response yet. However I found that Lichtenwalter and Lussier does not have a valid nd.edu email address. The thirds name Mr. Chawla is the name of a professor from Univ. of Notre Damme Indiana. I will search further in a hope to find a valid contact for the first names. However, despite the votes, I am still convinced that I am right this time. $\endgroup$ – rapaio May 19 '14 at 16:48

I would venture that it refers to regular random forests, but the author wants to bring out the distinction between a) the bagging / bootstrapping of the observations used for each tree and b) the random selection of a subset of the input parameters. not sure though.


As a colleague of the authors, I can address this question.

  • To directly answer the OP, @rapaio is correct: the top quotation means that the authors created 10 separate bags each with a random forest of 10 trees -- there will be 100 total trees.
  • As @rapaio mentioned, there's no clear cut reason why this performed better than 100 bags or 100 random forest. The Weka implementation made it easy to perform all five experiments experiments (single tree, 10 bags, 100 bags, 100 random forest, 10 bags of 10 random forests) and the authors felt the result was interesting enough to mention. Whether this is something that shows up as a general trend against other datasets (such as Kaggle or UCI) could be the basis for a good research paper.
  • @rapaio did not mention it, but there may also be some interplay between the various oob estimates and the final performance metric -- AUC (not accuracy). Perhaps this is a trend only observed when optimizing for rank-order and not necessarily discrete predictions.
  • $\begingroup$ I want to assure my understanding of the first point. By a single bag you mean a single Random Forest classifier, right? $\endgroup$ – Jack Twain Jul 13 '14 at 12:35
  • $\begingroup$ So basically you had 10 Random Forests classifiers, and then you take the majority vote of those 10 classifiers? $\endgroup$ – Jack Twain Jul 13 '14 at 12:37
  • $\begingroup$ That's correct. $\endgroup$ – DSea Jul 14 '14 at 11:25
  • $\begingroup$ Is it possible to get the datasets reported in the paper? I urgently need datasets that contain node attributes to test my methods. $\endgroup$ – Jack Twain Sep 9 '14 at 11:58

I tried to understand why bagging 10 random forests would work better than a random forest with 100 tress and I see no rational reason. I do not exclude that there could be some Weka implementation details.

However to answer your question I believe it is talking about a bagging ensemble with 10 bags and in each bag a random forest with 10 trees.

I do not have a proof, but there are some elements which I consider it provides enough evidence for this:

  • In the first paragraph, the last sentence which starts with "we found that neither 100.." it talks about building either a bagging with 100 trees or a random forest with 100 trees. Now 100 is 10 x 10, which are 10 bags x a random forest with 10 trees.
  • 10 bagged random forests means the same thing from the previous point (even if it is not usual to bag random forests)
  • in Weka is very easy to combine in this way some classifiers, since they can be chained together due to the fact that they implement the same interface (this is a programming stuff, I am a programmer, and I confirm that it is obvious that Weka intended to make such kind of experimenting as easy as possible)
  • there is a way to decrease the total compute out-of-the-bag error if you group 10 random forests with 10 trees into bagging, instead of 100 random forest. If you have 10 bags and in each bag one rf with 10 trees, than the oob error is computed for the other instance of the 10 bootstrap samples from the bags. These 10 oob samples sets are classified with a random forest instead of a normal random tree. Of course, the random forest has in general less variability, so the oob errors from the 10 bags could be lower than if you have the average of oob errors for 100 oobs predicted with trees. I do not know if this is a real improvement, it might be. From my point of view it is not, it looks like an introduction of a optimistic bias in oob estimation, but this is only my intuition and since I don't have an extensive experience, I might be wrong. Even so, it looks like a plausible track
  • last argument is a personal intuition: I asked myself why one would over-complicate these kind of formulations for nothing? I found reasonable to believe they did that with a purpose, and a "reasonable reason" is that they simply used a bagging of random forest.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.