1
$\begingroup$

I'm currently building a classification model in MS azure with the Two-Class Boosted Decision Tree algorithm. From my basic knowledge I know that the decision tree splits the features by a cut value in two or more leafs. Thus, when classifying new set of data, you just run through the tree according to the splits.

What I'm missing at this point is, how the probability of a certain outcome can be calculated in this classification process. When scoring the model with the test set, azure calculates the probability of each row of having a positive (=1) outcome.

Can someone explain me the association here between the decision tree with having rules assigning a clear outcome for every row but on the other hand probabilites are calculated for every row, which would not be not a clear outcome depending on the general cut-off value?

Thanks a lot!

This is part of the column in the scoring where probabilities for each row in the test set is calculated.

This is part of the calculated probabilities in azure.

$\endgroup$
0
$\begingroup$

Most classification decision tree algorithms provide both a class and probability. The probability that you see is the number of correct classifications at each leaf. So if at one leaf in your tree you have 1 observation of class A and 3 of class B you will see a probability of .75 for class B and the model will predict class B. You can read this slideshow for more details on decision trees http://www.cs.cmu.edu/afs/cs.cmu.edu/academic/class/15381-s06/www/DTs.pdf

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ But why is a probability calculated for every row that is being classified? Obviously in my model there's a cut-off value of 0.5, classifying rows with a higher probability than 0.5 as positive, while probabilities below 0.5 are negative. I get the usage of this idea for e.g. a logistic regression since its the underlying logic but I don't understand how probabilities can be calculated in a decision tree. For clarification, I added a screenshot of the column in azure that i'm struggling with. Maybe this helps. $\endgroup$ – TheDude May 10 '16 at 18:24
  • $\begingroup$ When you get down to the end of your decision tree, it more than likely is not perfect. Following the rules created by the tree in the trainig set perhaps 10 observations were positive and 90 were negative. Therefore, if a row in your test set follows the same rules it will tell predict the negative class with only 10% chance of it belonging to the positive class. $\endgroup$ – Brent Ferrier May 10 '16 at 18:35
  • $\begingroup$ I try to put that in my own words to what I understood. Please correct me if I'm wrong: The leafs of the tree don't necessarily have to contain rows of one class, in each leaf there could be rows from both classes. Depending on the ratio of the rows belonging to a class, there's a certain probability for each leaf that the tested row can be positive or negative, respectively. So for example if the training of a model results in a leaf with 4 positive labels and 1 negative label, then a tested row going down the decision tree to this leaf has a chance of 80% of being positive, right? $\endgroup$ – TheDude May 10 '16 at 18:53
  • $\begingroup$ yes, that is correct. The presentation linked goes over the construction of decision tress in great detail. $\endgroup$ – Brent Ferrier May 10 '16 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.