Skip to main content
Cross Validated

Return to Answer

added explanatino about splitting with missing values
Source Link
Simone
Simone
  • 7.1k
  • 3
  • 31
  • 53

Random forests are based on greedy induction of decision trees, which means that the best attribute to split on and best cut-off is computed independently for each internal node in the tree. Thus, nodes are not directly compared.

The article you cited does not specify that a split might be induced on a smaller number of points $N$ if an attribute has missing values. Indeed, in their experimental evaluation the adult dataset has missing values. It make sense to penalize split induced on attributes with missing values and they do that defining a new splitting criterion based on confidence intervals.

For example, let's say that you have the following data set:

A B Class
1 ? +
3 4 -
4 6 +

These 3 points can be splitted according to feature $A$ on the cut-offs 1 or 3, or according to feature $B$ on the cut-off 3. If we split according to feature $B$ we just take 2 points into account to compute the splitting criterion. If we split according to $B$ with the missing value ?, deciding whether to put the first point is a totally different story: C4.5 uses weights and CART uses surrogates splits.

Actually the topic of bias of splitting criteria has received attention in the past as well. The blog post you cited does not cite any previous work. We worked on a similar approach which penalizes missing values based on statistical significance rather than using confidence intervals: here. However, we focused in particular on categorical data sets which is another possible application domain. The positive side of using statistical significance is that this approach can be applied to the original Gini gain. Therefore it works also for multi-class classification and not just for binary classes.

Random forests are based on greedy induction of decision trees, which means that the best attribute to split on and best cut-off is computed independently for each internal node in the tree. Thus, nodes are not directly compared.

The article you cited does not specify that a split might be induced on a smaller number of points $N$ if an attribute has missing values. Indeed, in their experimental evaluation the adult dataset has missing values. It make sense to penalize split induced on attributes with missing values and they do that defining a new splitting criterion based on confidence intervals.

Actually the topic of bias of splitting criteria has received attention in the past as well. The blog post you cited does not cite any previous work. We worked on a similar approach which penalizes missing values based on statistical significance rather than using confidence intervals: here. However, we focused in particular on categorical data sets which is another possible application domain. The positive side of using statistical significance is that this approach can be applied to the original Gini gain. Therefore it works also for multi-class classification and not just for binary classes.

Random forests are based on greedy induction of decision trees, which means that the best attribute to split on and best cut-off is computed independently for each internal node in the tree. Thus, nodes are not directly compared.

The article you cited does not specify that a split might be induced on a smaller number of points $N$ if an attribute has missing values. Indeed, in their experimental evaluation the adult dataset has missing values. It make sense to penalize split induced on attributes with missing values and they do that defining a new splitting criterion based on confidence intervals.

For example, let's say that you have the following data set:

A B Class
1 ? +
3 4 -
4 6 +

These 3 points can be splitted according to feature $A$ on the cut-offs 1 or 3, or according to feature $B$ on the cut-off 3. If we split according to feature $B$ we just take 2 points into account to compute the splitting criterion. If we split according to $B$ with the missing value ?, deciding whether to put the first point is a totally different story: C4.5 uses weights and CART uses surrogates splits.

Actually the topic of bias of splitting criteria has received attention in the past as well. The blog post you cited does not cite any previous work. We worked on a similar approach which penalizes missing values based on statistical significance rather than using confidence intervals: here. However, we focused in particular on categorical data sets which is another possible application domain. The positive side of using statistical significance is that this approach can be applied to the original Gini gain. Therefore it works also for multi-class classification and not just for binary classes.

Source Link
Simone
Simone
  • 7.1k
  • 3
  • 31
  • 53

Random forests are based on greedy induction of decision trees, which means that the best attribute to split on and best cut-off is computed independently for each internal node in the tree. Thus, nodes are not directly compared.

The article you cited does not specify that a split might be induced on a smaller number of points $N$ if an attribute has missing values. Indeed, in their experimental evaluation the adult dataset has missing values. It make sense to penalize split induced on attributes with missing values and they do that defining a new splitting criterion based on confidence intervals.

Actually the topic of bias of splitting criteria has received attention in the past as well. The blog post you cited does not cite any previous work. We worked on a similar approach which penalizes missing values based on statistical significance rather than using confidence intervals: here. However, we focused in particular on categorical data sets which is another possible application domain. The positive side of using statistical significance is that this approach can be applied to the original Gini gain. Therefore it works also for multi-class classification and not just for binary classes.