In terms of classification task using decision trees, the formula for these looks almost the same. So, how are they different/same? what is the purpose of each in terms of impurity measure?

$\text{Entropy}~(p_1,p_2) = -\sum p_i \log (p_i); i= 1,2;$

$p_i $ are fractions. Say, if I have 2 Yes and 3 No in a node, $p_1=2/5$, $p_2=3/5$.

$\text{Deviance}~D= - 2\sum n_k \log(p_k) ;~k $ is the class in each leaf.

Both are used as impurity measures. But I am not able to understand the difference between these.

  • 1
    $\begingroup$ And can you provide the three formulas you are referring to (Entropy, Deviance, Impurity measure)? There are more then one. $\endgroup$ – Alecos Papadopoulos Nov 21 '13 at 22:35
  • $\begingroup$ Clarification please: This formulas relate to each leaf separately? If yes, I guess in each leaf we have "k" possible outcomes, each with count "n_k" in this leaf, and each with empirical relative frequency "p_k" in the specific leaf? If not, please clarify. $\endgroup$ – Alecos Papadopoulos Nov 23 '13 at 22:46
  • $\begingroup$ Yes, that is right. $\endgroup$ – leviathan Nov 24 '13 at 7:17

They are same. It's a nomenclature difference among authors. Gini is different though. Using your notation it would be $1 - \sum p_i^2$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.