11
$\begingroup$

Can someone please explain to me when to use Gini impurity and information gain for decision trees? Can you give me situations/examples of when is best to use which?

$\endgroup$
9
$\begingroup$

You should try them both as part of parameter tuning.

Theoretically Gini impurity minimizes Brier score while entropy/information gain minimizes log loss so which of those you're interested in makes some difference. However other things like how likely each is to discover multivariate effects in greedy tree growth instead of getting "distracted" by univariate ones that also play into things. Ie you may get better generalization from an impurity metric that doesn't always select the "best" split.

In practice (in the context of rf, more then cart) i've found entropy works better for cleaner low dimensional data sets where you're trying to fit a more complex signal as well as possible while gini works better for noisy, highly dimensional ones where your trying to uncover a simple signal from amongst many noisy potential signals. This is just my experience though and will almost certainly not hold in all cases.

Note: started as a comment but deleted and moved to an answer to format an expand on things.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.