4
$\begingroup$

I am a bit confused, as it seems like my teacher has told me something different than what I see and hear on YouTube videos.

My problem is as follows:

My teacher told us to that the probability when deciding the root of the decision tree needs to be as close to 50% as possible. So, let's say we have this table:

enter image description here

My teacher told me for the ID3 algorithm, we can "clearly see, without having to calculate the information gain, that we need to pick "Growling" as our root as this is 50%, hence the 4 times YES and 4 times NO"

Well, okay, but why? When I check YouTube videos, they say that we need to take the most pure ones, so the ones where we have the most difference (like: 7 times YES and 0 times NO), which is certainly not "growling", as it's impure...

As you can see I am confused. Is this 50% strictly for the ID3 algorithm or is there something I am missing here? Could someone help me out, please?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

ID3 algorithm would first takes growling because creates two nodes that are overall purer than the root: root has 3 yes and 5 no, if you use growling to split your data you'll have 2 yes and 2 no in one node (that can be splitted using smelly) and 1 no and 3 yes in the other node. the first node gives you an information loss, the second one an information gain, the algorithm calculates the mean of the two measures, and finding it positive, performs the split. information gain (loss is his negative) is based on entropy, which measures the heterogeneity of the response variable in the nodes.

Improve this answer
$\endgroup$
6
  • $\begingroup$ But does this mean that you first need to check what comes AFTER you would hypothetically choose some root? I mean, do you choose the root here because it's 2 YES and 2 NO or because of the possibilities after you would choose this? $\endgroup$
    – Siyah
    Commented Mar 21, 2017 at 19:01
  • $\begingroup$ you choose the variable for performing the split for the information gain that you will get choosing it, and that's calculeted on nodes created by the split, so, yes. $\endgroup$
    – carlo
    Commented Mar 21, 2017 at 19:07
  • $\begingroup$ Okay, but does this mean that the information gain of an element with 2 yes and 2 no is higher than 3 yes 5 no? $\endgroup$
    – Siyah
    Commented Mar 21, 2017 at 19:18
  • $\begingroup$ no, entropy of a node wtih 2 yes and 2 no is higher than one with 3 yes and 5 no, so there's an information loss. however, this is averaged with the information gain from the other node, which has a lower entropy, and the overall information gain is positive $\endgroup$
    – carlo
    Commented Mar 21, 2017 at 23:43
  • $\begingroup$ You say no, but you also state that a node with 2 yes and 2 no is higher... higher what? Higher information gain? If so, why do you say no? Can you be more concise / precise please? $\endgroup$
    – Siyah
    Commented Mar 22, 2017 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.