# Classification trees - how to avoid “cherry picking”?

This post How to compare the performance of two classification methods? (logistic regression and classification trees) represents a very similar problem I am just trying to solve - I have a dataset based on survey (arguably with less then desirable design), with about 3000 observations, and 100+ categorical variables. The variable of interest is binary. Despite identifying over 30 variables as significant, logit regression does not provide very interesting increase in the correctly predicted cases (approx. in 80% y=1, 20% y=0), the overall increase in the number of correctly predicted cases is roughly from 80% (naive model) to 81.5%.

Which brings me to classification trees. While very appealing as a method, they inherently do not provide very stable results (and the choice of parameters influences the output). Using several classification methods you might obtain somewhat similar (i.e. several variables can be in all trees, but what about the variables that show up only in one tree?)

Which brings me to a question, if there are somewhat similar, but distinct outputs, how do you interpret the results?

I used rpart, evtree, ctree and chaid (because chaid can only work with categorical variables, all variables in the example are categorical to allow direct comparison), but I am not much wiser in terms of what the data say.

I would attach a real dataset, but for the sake of discussion, let's create a random one. The variable of interest is binary variable y

Df1 <- data.frame(
y = as.factor(sample(0:1,300,r=T)),
x1 = as.factor(sample(1:8,300,r=T)),
x2 = as.factor(sample(1:3,300,r=T)),
x3 = as.factor(sample(1:2,300,r=T)),
x4 = as.factor(sample(1:8,300,r=T)),
x5 = as.factor(sample(1:3,300,r=T)),
x6 = as.factor(sample(1:2,300,r=T)),
x7 = as.factor(sample(1:8,300,r=T)),
x8 = as.factor(sample(1:3,300,r=T)),
x9 = ordered(sample(1:5,300,r=T))
)
library("partykit")
library("rpart")
library("evtree")
library("CHAID")

ct <- ctree(y~ . , data = Df1, minbucket=50)
plot(ct)
rp <- rpart(y ~ .,data=Df1, minbucket=30)
plot(as.party(rp))
ev <- evtree(y ~ ., data = Df1, maxdepth = 5)
plot(ev)
ctrl <- chaid_control(minsplit=90, minbucket=30, minprob=0.05,alpha2=0.01, alpha3=-1, alpha4=0.01)
chaid1 <- chaid( y ~ ., data= Df1, control=ctrl)
plot(chaid1,cex=0.6)