I have a model which generates probabilities. I pick an optimal threshold for the purposes of classification, by choosing the probability cutoff so that the percentage of cases misclassified is minimised.

I am wondering about the relationship between this threshold and the splitting rule used in a classification tree with one split, such that the root node is split using these probabilities.

If the splitting rule is gini diversity index, the split point of the tree is not the optimal threshold. I have verified that using the optimal threshold for the split point actually results in less information gain.

Why is this happening? Would the split point be the optimal threshold if misclassification rate was used as the splitting rule instead?

  • 1
    $\begingroup$ It is only valid to compute a threshold if you plug the utility/loss/cost function in and this function is the same for every individual. Thresholds cannot come from the data themselves. Any attempt to derive a threshold form the data themselves is assuming a strange utility function. $\endgroup$ – Frank Harrell Oct 28 '15 at 0:33
  • $\begingroup$ I'm not sure whether this question is about choosing a cutoff for class assignment on the basis of model-generated probabilities, or if you believe the splits in your decision tree are sub-optimal. Could you please clarify this? $\endgroup$ – Sycorax Oct 28 '15 at 0:35
  • $\begingroup$ Let me see whether I understand your comment @FrankHarrell, correct me if I am wrong: In the case of thresholding the probabilities generated from the model, the cost function for an individual is 1 if the prediction is wrong and 0 if the prediction is correct. You are saying that I cannot expect a classification tree to give a threshold as the cost function is ill defined. Yet, the the classification error seems (to me) to be the same cost function as used for thresholding the probabilties. So wouldn't using classification error to split the tree derive a split equal to the optimal threshold? $\endgroup$ – Alex Oct 28 '15 at 0:37
  • $\begingroup$ @user777, I have made some edits to the question. Essentially my query is about why the split value for a classification tree does not equal to optimal threshold for the probabilistic model. $\endgroup$ – Alex Oct 28 '15 at 0:39
  • $\begingroup$ @Alex don't confuse $Y$ with the cost/utility function, and are assuming the utility function is the same for every subject. With a well-calibrated estimate of the probability of the outcome, and the utility function, the risk action threshold is set. Example: the cost of sending mail to a potential customer is \$1 and the profit from a purchase is \$100. A probability of 0.01 of responding to the ad is the breakeven point for the expected profit. A lift curve where you rank potential customers on predicted probability and send mail until the money runs out is also a good approach. $\endgroup$ – Frank Harrell Oct 28 '15 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.