-1
$\begingroup$

I would like to compute a ROC-Curve for a decision tree created with sklearn based on the CART algorithm. Logically, if I compute it with maximum depth I always get a probability of 1 or 0 (discrete classifier) for a classification. I red https://datascience.stackexchange.com/questions/323/how-can-we-calculate-auc-for-a-simple-decision-tree on how to get probabilities from the ration in the end node, but this would mean, that I would have to crop the tree, by limiting the depth. But its often those last decisions that can be crucial for a correct classification. Therefore I would like to know, if there is a way to create a score or probability for a maximum depth decision tree, (for example by taking into account the number or previous nodes or the summarized entropy or the fraction of the training data in that node)

$\endgroup$
  • 1
    $\begingroup$ You really need to just fit a shallower tree, as what you've described is overfitting. Im not sure what you mean by "But its often those last decisions that can be crucial for a correct classification". That may be true through the lens of the training data, but on test data those final nodes are just overfitting. $\endgroup$ – Matthew Drury Sep 13 '17 at 16:54
  • 1
    $\begingroup$ The decision thresholds involved in ROC analysis need not be probabilities. Which do you want, a ROC curve or probabilities? $\endgroup$ – Kodiologist Sep 13 '17 at 17:58
  • $\begingroup$ @Kodiologist I'd like to compute a ROC curve, therefore I need some kind of 'goodness' score (could be probabilities) as a threshold, that's why I asked if someone has an idea to evaluate the decision. $\endgroup$ – Paul Zierep Sep 14 '17 at 8:23
  • $\begingroup$ @MatthewDrury Your argument seems reasonable, I will continue with a shallower tree, but I would still like to know (out of interest), if and how a maximum depth tree could be scored (maybe by taking the number of nodes into account and thereby showing the over-fitting). $\endgroup$ – Paul Zierep Sep 14 '17 at 8:33
1
$\begingroup$

You're trying to have your cake and eat it, too.

Logically, if I compute it with maximum depth I always get a probability of 1 or 0 (discrete classifier) for a classification. I red https://datascience.stackexchange.com/questions/323/how-can-we-calculate-auc-for-a-simple-decision-tree on how to get probabilities from the ration in the end node, but this would mean, that I would have to crop the tree, by limiting the depth. But its often those last decisions that can be crucial for a correct classification.

A tree that has probabilities in the leafs compute the probability by taking the proportion of each class in that leaf. If you want pure leafs, then your probabilities are going to be $\mathbb{P}(class = 1)=1$ or $\mathbb{P}(class = 1)=0$ by construction. If you want predicted probabilities that are greater than 0 and less than 1, then you need impure leafs. You can't have both.

If you're willing to use an ensemble of trees, then you have more flexibility. Random forests, for example, take the average predictions over randomized trees trained on randomized data, so predicted probabilities can be found by averaging over binary votes of fully-grown trees.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.