Which is a better cost function for a random forest tree: Gini index or entropy?

I am trying to implement random forest in Clojure.


As I found in Introduction to Data Mining by Tan et. al:

Studies have shown that the choice of impurity measure has little effect on the performance of decision tree induction algorithms. This is because many impurity measures are quite consistent with each other [...]. Indeed, the strategy used to prune the tree has a greater impact on the final tree than the choice of impurity measure.

Therefore, you can choose to use Gini index like CART or Entropy like C4.5.

I would use Entropy, more specifically the Gain Ratio of C4.5 because you can easily follow the well-written book by Quinlan: C4.5 Programs for Machine Learning.

  • 3
    $\begingroup$ Little remark -- entropy uses logs, what can be a computational time issue. $\endgroup$ – user88 Dec 24 '11 at 9:43
  • 8
    $\begingroup$ That remark is about pure decision trees, not random forests though. You don't usually prune a tree in a random forest because you're not trying to build a best tree. So it seems misleading to talk about what's more important: pruning or impurity measure. The goal is to find the best tree to use with random forest. $\endgroup$ – Chan-Ho Suh Nov 9 '13 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.