Which is a better cost function for a random forest tree: Gini index or entropy?
I am trying to implement random forest in Clojure.
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Gini and Entropy are not cost function but they are the measures of impurities at each node to split the branches in Random Forest. MSE(Mean Square Error) is the most commonly used cost function for regression. Cross Entropy cost function is used for classification. $\endgroup$– Kans AshokCommented Oct 10, 2020 at 12:09
-
1$\begingroup$ <rant>The "Gini index" here is another example of machine learning confusing a term which already exists meaning something else for a concept which already has several existing names. The other use of "Gini index" as an inequality measure for income or wealth (related to the Lorenz curve) is better known generally, while this ML measure is also essentially what is known in different fields as the Herfindahl–Hirschman Index, the Simpson Index, the Blau Index, the Hunter–Gaston index, the inverse participation ratio, the discriminatory power, or the effective number</rant> $\endgroup$– HenryCommented Oct 10, 2020 at 14:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
As I found in Introduction to Data Mining by Tan et. al:
Studies have shown that the choice of impurity measure has little effect on the performance of decision tree induction algorithms. This is because many impurity measures are quite consistent with each other [...]. Indeed, the strategy used to prune the tree has a greater impact on the final tree than the choice of impurity measure.
Therefore, you can choose to use Gini index like CART or Entropy like C4.5.
I would use Entropy, more specifically the Gain Ratio of C4.5 because you can easily follow the well-written book by Quinlan: C4.5 Programs for Machine Learning.
-
3$\begingroup$ Little remark -- entropy uses logs, what can be a computational time issue. $\endgroup$– user88Commented Dec 24, 2011 at 9:43
-
9$\begingroup$ That remark is about pure decision trees, not random forests though. You don't usually prune a tree in a random forest because you're not trying to build a best tree. So it seems misleading to talk about what's more important: pruning or impurity measure. The goal is to find the best tree to use with random forest. $\endgroup$ Commented Nov 9, 2013 at 20:51