Information gain is defined as $$IG(T, a) = H(T) - H(T|a),$$ where $H(T|a)$ is the conditional entropy of $T$ given attribute $a$, and $H(T)$ is the prior entropy of our dataset before we test out the split on attribute $a$.

Many decision trees use $IG$ as the splitting criteria to determine which attribute to use for the split at a given node, where we choose the attribute $a$ that maximizes $IG(T, a)$.

However, it seems to me unnecessary to use the whole equation. For a given node, $H(T)$ will remain constant regardless of the attribute that we choose, so $$\begin{align}a_{best} &= \underset{a\in A}{\operatorname{argmax}}H(T) - H(T|a) \\ & = \underset{a\in A}{\operatorname{argmax}} - H(T|a) \\ & = \underset{a\in A}{\operatorname{argmin}} H(T|a) \end{align}$$

I'm wondering why we talk about information gain, as opposed to just minimizing the conditional entropy. I know finding the prior entropy isn't very computationally expensive, so just minimizing conditional entropy wouldn't be a big performance improvement, but I'd like to know if I'm misunderstanding something, or perhaps if there's a historical or theoretical reason behind using $IG$ as defined.

  • 1
    $\begingroup$ I don't think there is a difference - neither did my superviser nor censor correct me when I wrote, that "we seek to minimize H(T|a) to find the optimal split" in my master's thesis. Either they did not read it through or they agreed. The idea behind IG might be if you have some non-linear scale/weight-whatever, but I cannot come up with an example of that $\endgroup$
    – CutePoison
    May 10, 2020 at 8:45


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.