When splitting attributes while constructing a random tree, I use information gain in order to determine the best value to split the tree on. I add nodes to the tree until a stopping criterion is met. What is the minimum value of information gain, to be used as a stopping criterion?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The minimum information gain required for a split is a tunable parameter and probably should be determined using cross validation on a problem by problem basis. You can also run statistical significance tests for each split, addressing the question whether the split provides a statistically significant increase in information gain over a random split. Check out pages 12 and 13 from this PDF for an example of statistical tests on splits.
answered May 30, 2013 at 13:09
$\begingroup$
$\endgroup$
1
I agree with
The minimum information gain required for a split is a tunable parameter
The most important factor to consider while tuning this parameter is bias-variance tradeoff:
- If the minimum information gain for stopping criteria is very low, the depth of the tree will be very high and consequently, the number of samples at each leaf node will be low. This is a high variance situation (over fitting).
- If the minimum information gain for stopping criteria is very high, the depth of the tree will be very low and consequently, the number of samples at each leaf node will be high. This is a high bias situation (under fitting).
-
$\begingroup$ For each bullet point, when you say "This is a XXX situation (YYYY)" do you mean "This would cause a XXX situation (ie. YYYY)"? $\endgroup$ Commented Jun 16, 2021 at 4:25