I want to predict a categorical variable using also categorical predictors. Currently, I am looking at classification and regression trees (CART).
The prediction quality is "good enough", except for the presence of impossible combinations. In the following minimal example, the combination a==2, b==2 is impossible, yet the estimation decides not to use b for splitting.
> library(rpart)
> d <- data.frame(a=rep(factor(c(1,1,2)), 100000), b=factor(c(1,2,1)))
> #d
> xtabs(~., d)
b
a 1 2
1 1e+05 1e+05
2 1e+05 0e+00
> (tr <- rpart(a~b, d))
n= 300000
node), split, n, loss, yval, (yprob)
* denotes terminal node
1) root 300000 1e+05 1 (0.6666667 0.3333333) *
When simulating stochastically from this model (by choosing the leaf value by sampling using the annotated probability vector, here $(2/3, 1/3)$, as weights), the combination 2, 2 will occur:
> prob.m <- predict(tr, d, type="prob")
> d$a.sim <- apply(prob.m, 1, function(x) sample.int(length(x), size=1, prob=x))
> xtabs(~a.sim+b, d)
b
a.sim 1 2
1 133041 66615
2 66959 33385
Is there a way to avoid this, perhaps using another method?
This is just a small example for a more general case. I have around 10 predictors, and I want to exclude all combinations of two (or perhaps three) attributes that have no observation in the sample.
I am aware of the "loss matrix" that can be specified as a parameter to rpart, but this is prohibitive if many predictors are used.