Can someone please explain to me when to use Gini impurity and information gain for decision trees? Can you give me situations/examples of when is best to use which?
$\begingroup$
$\endgroup$
1
asked Dec 23, 2014 at 20:18
-
$\begingroup$ This answer on DS.SE also mentions (1) computational simplicity of Gini impurity and (2) the availability of a closed-form optimum for Gini impurity. $\endgroup$– paperskilltreesCommented Apr 10, 2023 at 19:42
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You should try them both as part of parameter tuning.
Theoretically Gini impurity minimizes Brier score while entropy/information gain minimizes log loss so which of those you're interested in makes some difference. However other things like how likely each is to discover multivariate effects in greedy tree growth instead of getting "distracted" by univariate ones that also play into things. Ie you may get better generalization from an impurity metric that doesn't always select the "best" split.
In practice (in the context of rf, more then cart) i've found entropy works better for cleaner low dimensional data sets where you're trying to fit a more complex signal as well as possible while gini works better for noisy, highly dimensional ones where your trying to uncover a simple signal from amongst many noisy potential signals. This is just my experience though and will almost certainly not hold in all cases.
Note: started as a comment but deleted and moved to an answer to format an expand on things.
