In section 7 of the paper Random Forests (Breiman, 1999), the author states the following conjecture: "Adaboost is a Random Forest".
Has anyone proved, or disproved this? What has been done to prove or disprove this post 1999?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ Please read stats.stackexchange.com/questions/77018/… Maybe you'll find your answer there $\endgroup$– user75008May 1, 2015 at 6:11
-
$\begingroup$ @user75008 Thanks! So, section 7 provides another conjecture, such that if proven, shows that adaboost is equivalent to random forest. Has anyone shown this conjecture to be true? $\endgroup$– AlexMay 1, 2015 at 6:15
-
$\begingroup$ @user75008 I am reading your link, stats.stackexchange.com/questions/77018/…, do you think it suggests that Adaboost is not equivalent to Random Forest? $\endgroup$– AlexMay 1, 2015 at 6:26
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Interesting question. A bunch of work on explaining ada boost via a few different tactics has been done since then.
I did a quick literature search and this somewhat odd paper appears to be the most recent one on the subject and also reviews a bunch of the intercedent work by Leo Breiman and others:
http://arxiv.org/pdf/1212.1108.pdf
I have no idea if their results are valid but they claim to have failed to prove Breiman's conjecture but to have proved a weakened version of it claiming adaboost is measure preserving but not necessarily ergodic.
They also present some empirical evidence that adaboost does in fact sometimes overfit.
I think that suggests adaboost may be related to a random forest but not entirely (or not always) equivalent in the way Breiman conjectured?
-
$\begingroup$ thanks, so I guess this is still an open question, but your last statement is telling. $\endgroup$– AlexMay 6, 2015 at 6:05
-
1$\begingroup$ Yeah I think it is still open. I also think that interest has dropped off in analyzing AdaBoost as [stochastic] gradient boosting machines have become more popular. AdaBoost is a form of gradient descent (en.wikipedia.org/wiki/AdaBoost#Boosting_as_Gradient_Descent) and thinking in terms of explicitly randomized gradient descent may be more intuitive and more practical then the equivalency Brieman proposed. (Ie even if it were true it might be really hard to sample from the needed distribution in practice.) $\endgroup$ May 6, 2015 at 14:46
-
$\begingroup$ I just saw this new paper on the subject: arxiv.org/pdf/1504.07676v1.pdf $\endgroup$ May 8, 2015 at 15:33
-
$\begingroup$ Very interesting if true! "We conclude that boosting should be used like random forests: with large decision trees and without direct regularization or early stopping." $\endgroup$– AlexMay 8, 2015 at 23:39
-
$\begingroup$ @RyanBressler Your comment is timestamped "May 8, 2015" but the cited article is (arxiv.org/pdf/1504.07676v1.pdf) is from 2022. Strange? $\endgroup$– ThereMay 30, 2022 at 13:11