Model comparison

Can anyone suggest the statistical tools to compare CART, conditional inference tree, and random forests? I use these three algorithm for regression analysis and want to choose the better one.

-
Are you talking about traditional model comparison techniques, along the lines of AIC and BIC? – Kyle. Oct 31 '12 at 0:49

I recommend using some form of cross-validation. It's even the name of the site! 10-fold CV is commonly used, but there are other, more sophisticated methods.

Here is an example, based on the caret vignette:

#Load Data
set.seed(42)
require(mlbench)
data(BostonHousing)
y <- BostonHousing[,14]
X <- BostonHousing[,1:13]

#Use the same CV-folds for each model
require(caret)
myControl <- trainControl(method='cv', number=10, index=createFolds(y, k=10))

#Fit models
model_rpart <- train(X, y, method='rpart', trControl=myControl)
model_ctree <- train(X, y, method='ctree', trControl=myControl)
model_rf <- train(X, y, method='rf', trControl=myControl)

#Plot
resamples <- resamples(list(
rpart=model_rpart,
ctree=model_ctree,
model_rf=model_rf
))
dotplot(resamples, metric='RMSE')


In this example, the random forest has a lower cross-validated RMSE, so I would select that model.

-