When splitting attributes while constructing a decision tree, i can use information gain or information gain ratio to try and determine the best value to split the tree on.
- I'd use information gain when I have smaller numbers of distinct values (mostly nominal and ordinal values that are not continuous)
- I'd use information gain ratio for attributes which have large numbers of distinct values (mainly continuous values)
That said, do I necessarily have to choose one method for use with all attributes across the entire tree?
My gut feeling is no, because information gain or information gain ratio for comparison to other attributes is not a relevant comparison; it's only relevant for comparison to a different value of the same attribute for the purposes of producing a split.
I'm thinking that one might be able to go so far as to swap them for different splits in the tree, depending on the training data. If the subset of values for that attribute is small compared to previous splits where the distinct values were large, one could use the value that's more biased to producing a more useful result.
entropytag because entropy is a fundamental part of information gain (granted, I believe Gini impurity could be used too). If it's inappropriate to the question feel free to edit it out. I would have addedinformation-gainbut it doesn't seem to exist and I currently don't have the rep to create it. – casperOne Sep 1 '12 at 14:17