4
$\begingroup$

one approach to deal with the unbalanced dataset is to choose the models that can hand this type of dataset well such as decision tree, but why decision tree can handle the unbalanced dataset well?

$\endgroup$

2 Answers 2

5
$\begingroup$

Decision trees do not always handle unbalanced data well.

If there is relatively obvious particular partition of our sample space that contains a high-proportion of minority class instances, decision trees can probably find it but that is far from a certainty. For example, if the minority class is strongly associated with multiple features interacting with each other, it is rather demanding for a tree to recognise the pattern; even if it does, it will probably be a rather deep and unstable tree that will be prone to over-fitting; pruning the tree will not immediately solve the problem because it will directly affect the ability of the tree to utilise those interactions.

Generalisations of decision tree algorithms as random forests and gradient boosting machines offer a much better alternative in terms of stability without sacrificing any performance. Similarly using a GAM with an interacting spline can also provide another potentially viable alternative.

$\endgroup$
3
  • 2
    $\begingroup$ Absolutely agree that decision trees don't inherently solve the class imbalance problem. The optimizing function itself will have a large impact - if your decision tree is trained to optimize accuracy, for example, it will handle class imbalance much worse than a decision tree that optimizes the F1 score or AUC. $\endgroup$ Feb 25, 2020 at 21:47
  • $\begingroup$ @NuclearWang: I agree (+1); I assume that the "standard decision tree" is the CART so we use Gini impurity by default. That said, even if we employ a "good measure" we might still over-fit and experience poor generalisation. $\endgroup$
    – usεr11852
    Feb 25, 2020 at 22:01
  • $\begingroup$ @NuclearWang The splitting process is always done through gini or entropy and therefore always maximises accuracy - the only way you can influence F1 or AUC on a single tree would be through finding the subtree that optimizes those via postpruning, or to choose the hyperparameters of the ensambling procedure. Therefore yes - standard trees do not handle unbalanced classes, but some optimization around can. $\endgroup$
    – Davide ND
    Feb 26, 2020 at 15:44
3
$\begingroup$

I would like to add something to the previous answer - though it's already good (+1).

Decision trees implementations normally use Gini index or Entropy for finding splits. These are functions that are maximized when the classes in a node are perfectly balanced - and therefore reward splits that move away from this balance. This means that the splits are always done assuming that the classes distribution is $1/K$ (or 50-50 in the binary case), and this is particularly clear when classes are VERY unbalanced. In those cases, since classification is done by majority voting, most of the leaf nodes will contain majority class elements and the performance will be sub-optimal.

There are a number of papers that discuss this issue, but I really suggest reading Using Random Forest to learn Imbalanced Data, that proposes the use of Weighted Gini (or Entropy) to take into account the class distribution, or using a mixture of Under and Over sampling of the classes when bagging decision trees.

Indeed, standard trees are mathematically not constructed to deal particularly well with unbalanced data, and some adjustments are needed (to the voting, splitting, or sampling) - which is also why many implementations allow the use of class weights.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.