I have a rather unbalanced data set and want to use rpart to build a classification tree. After building the full tree, I prune it back using the 1-SE rule. On average, only 1-2 splits are suggested. I was wondering, as far as I understood, cp is basically picked to maximise accuracy. However, this is suboptimal since accuracy is not a good performance measure for unbalanced data sets. My initial plan was basically to adjust the classification threshold eventually and pick the one which maximises for example the F1-score. But this is somehow inconsistent in my view. Running a tree first and pick cp so that accuracy is maximised although I know that cannot be the ultimate goal here. I would really appreciate some help here? I'm a little bit stuck here. Thank you!

  • $\begingroup$ What about using svm with different class weights? I am having a similar issue where only 0.06% for my data are in class 1. The method you proposed would make sense if you use cross-validation/bootstrapping to optimize the cutoff. $\endgroup$ – Matthew Lau Dec 18 '15 at 18:44
  • $\begingroup$ Thank you for the comment! I am using cross-validation. What do you mean exactly by optimising the cutoff? But the question remains, rpart is trained to optimise accuracy even though this is not optimal here. Or am I missing something? $\endgroup$ – Patrick Balada Dec 18 '15 at 18:54
  • $\begingroup$ I am actually not too sure about that too .. but what I was thinking is maybe manually write a function in r to get the cp cutoff that maximizes F1-score. $\endgroup$ – Matthew Lau Dec 18 '15 at 18:59
  • $\begingroup$ Interesting thought - let me please know when you find a solution. My current approach is to use a loss matrix to weight misclassifications differently. This gives me pretty high balanced accuracy. However, I'm not sure that's the way to go... $\endgroup$ – Patrick Balada Dec 18 '15 at 22:01
  • $\begingroup$ I am surprised to see that approach inproved accuracy, normally when you increase the loss portion for one(e.g. False Positive), it will also the occurrence of the other (False Negative). $\endgroup$ – Matthew Lau Dec 18 '15 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.