1
$\begingroup$

I read the statement, listened to a coursera lecture that Ensemble techniques like ADA Boost and Random Forest are better and used more often than pure Decision Trees.

I am right to assume for Classification that therefore ID3, CART et al approaches no longer prevail? And that we learn these as part of courses only and in practice a pure decision tree like CART is no longer applied?

$\endgroup$
  • $\begingroup$ ADA Boost is only for binary classifications so I learnt. $\endgroup$ – thebluephantom May 14 at 10:41
0
$\begingroup$

First you should understand that why we moved to Random forest or AdaBoost.

We always want our model to have low bias and low variance. Suppose you train a pure decision tree which has low-bias and high-variance, so now if we use Random Forest then we will get a low-variance in the model with the almost same bias(because of Bootstrap technique in random forest).So here in this case Random Forest is a good option to use instead of pure decision tree to achieve low variance in the model.

Now let suppose you train a very shallow decision tree which results in high-bias and low variance model. Now using the AdaBoost you can get a low-bias and low-variance model(because off additive weight technique in AdaBoost). So in this case AdaBoost is a good option instead of pure decision tree to achieve low bias in the model.

So it depends on the bias and variance of the model you are training. If your pure decision tree is already giving you a low-bias and low-variance model then there may not be much significant improvement over using either Random Forest and AdaBoost. Random Forest and AdaBoost are techniques to reduce the variance and bias in the model respectively.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ OK. Not up to RF yet. But, I got that issue on shallow decision tree (hb, lv) from the course --> stumps. I then look at your 2nd para & 3rd para and they are correct but a little non-committal in terms of the question. The lecturer stated these boosting approaches are used by and large. So, my question is not entirely answered . $\endgroup$ – thebluephantom May 10 at 20:20
  • $\begingroup$ So what do you think? $\endgroup$ – thebluephantom May 11 at 8:07
  • $\begingroup$ Yes they are used by and large because obviously they will perform better then base model of decision tree. But you should always consider the cost at which you achieved this increased in performance. Suppose you have only got 1% increase in performance and used good number of hours to train the model then it is not necessary to use AdaBoost. But if you only care about the performance then you should definitely use random forest or AdaBoost! $\endgroup$ – Deepak Chaudhary May 11 at 15:05
  • $\begingroup$ That said, computing is cheap. $\endgroup$ – thebluephantom May 11 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.