Is there an ANOVA table generalization for two nested CART models?

The last line is an example of what I'm looking for:

data(airquality)
attach(airquality)

lm1 <- lm(Ozone ~ Solar.R+Wind)
lm2 <- lm(Ozone ~ Solar.R+Wind+Temp)
anova(lm1 , lm2)

require(rpart)
rp1 <- rpart(Ozone ~ Solar.R+Wind)
rp2 <- rpart(Ozone ~ Solar.R+Wind+Temp)
anova(rp1 , rp2) # this doesn't exist - is there something like it? some sort of anova.rpart function?


Thanks!

-
What do you need the table for? Are your trying to compare the 2 models? Are you trying to combine the 2 models? –  Zach May 25 '11 at 20:04
Hi Zach - In short - I would like to compare and get a P value –  Tal Galili May 25 '11 at 20:24

Just to add to Frank's points and paint a somewhat finer picture: CART/RPART is indeed highly exploratory and adding a p-value is difficult. I have seen some rare cases where people tried to use bootstrapping to get such p-value but I agree with Frank that it's not worth the effort.

As for combining statistical inference with recursive partitioning more generally: The CTree and MOB methods implemented in the "party" package as well as several other tools outside R (especially the work of Loh & co-authors) tries to combine standard statistical tests (nonparametric in case of CTree, parametric for MOB) for growing the trees. They also control their error level in the sense of a closed testing procedure. However, the inference for the final fitted tree is still not trivial. The situation is similar to a regression model where you have selected interactions between your regressors in a forward search (and we know how Frank feels about that). Hence, also the "party" package has no anova() methods. We do provide AIC() for "mob" objects, though. It is not strictly valid (because we haven't done full optimization of the log-likelihood but only forward search) but it would be conservative. So the error is into the preferable direction.

-
Hello Achim, I'm happy to see you join this site :) After reading what you've written it sounds like my best bet (for the context of the problem I'm currently investigating) to go with bootstrapping (or some randomization test). You said you've seen such rare cases, if you can help me find what they have done I'd be glad to see it. Otherwise I guess I'll try to write something like this myself. Thanks again, Achim, for your help. –  Tal Galili May 26 '11 at 7:59
I would have to Google this myself. I haven't got any specific publication for this in mind. The idea would essentially be to draw samples from the "smaller" model, e.g., in a model-based fashion using a normal distribution within each leaf or so. And then fit the "larger" model on many such artificial data sets. –  Achim Zeileis May 26 '11 at 15:03
...and a further comment: Similar strategies are also used for computing so-called variable importance measures, especially in random forests (but also in single trees). I also wouldn't recommend these for formal inference but for peeking into the black box they are often quite useful. See biomedcentral.com/1471-2105/8/25 and biomedcentral.com/1471-2105/9/307 and the references therein for some pointers. These are also available in "party". –  Achim Zeileis May 26 '11 at 15:07
Thank you for all the references Achim, I will have to go through them in more detail and see what I can come up with. Thanks again! –  Tal Galili May 26 '11 at 18:45