How do I know which decision tree model is better?

Question

I built a few simple decision trees and produced 3 models. The 1st model $m_1$ was completed without any modifications. The 2nd model $m_2$ had a reduced minsplit of 2. The 3rd model $m_3$ was based on the 2nd model but pruned at the minimum cp.

Here are the confusion matrices for the 3 models:

> print(cm1)
     
pred  No Yes
  No  14   2
  Yes  3  11

> print(cm2)
     
pred  No Yes
  No  17   0
  Yes  0  13

> print(cm3)
     
pred  No Yes
  No  16   2
  Yes  1  11

Here are the cptables for m2 and m3:

>m2$cptable
          CP nsplit rel error    xerror      xstd
1 0.61538462      0 1.0000000 1.0000000 0.2087816
2 0.05128205      1 0.3846154 0.6923077 0.1930754
3 0.03846154      4 0.2307692 0.6153846 0.1863169
4 0.00000000     10 0.0000000 0.7692308 0.1986145

> m3$cptable
          CP nsplit rel error    xerror      xstd
1 0.61538462      0 1.0000000 1.0000000 0.2087816
2 0.05128205      1 0.3846154 0.6923077 0.1930754
3 0.03846154      4 0.2307692 0.6153846 0.1863169

From these tables, my main question is: which model is better and what comparison parameters should I use to compare the models?

Based on the tables alone I would use $m_2$ as it does not give me any errors. However, $m_3$ should be a better model as it was pruned at the minimum cp value. But $m_3$ actually has a higher error rate here despite having a lower x-error.

Any advice would be great!

Answer 1

Depending on the number of candidate predictors, the sample size needed for a single tree method to be reliable is in the range of n=100,000. You also seem to be doing forced-choice classification which means that you are applying a hidden utility function that will likely lead to poor decisions and ignoring gray zones. Methods that use an ensemble of trees (bagging, boosting, random forest) exist because single tree methods are not competitive for prediction.

Answer 2

I would suggest you to check the AUC-ROC Curve for the three models and see which one is yielding the highest result.

These values come in handy when we fail to select a model on the basis of only the confusion matrix or if there is a huge imbalance between the two classes (which there isn't in your case though). So check it out and let me know in the comment how they are coming. A tutorial to do that in R is given here: https://www.r-bloggers.com/2016/11/calculating-auc-the-area-under-a-roc-curve/