Model comparison

Question

Can anyone suggest the statistical tools to compare CART, conditional inference tree, and random forests? I use these three algorithm for regression analysis and want to choose the better one.

Answer 1

I recommend using some form of cross-validation. It's even the name of the site! 10-fold CV is commonly used, but there are other, more sophisticated methods.

Here is an example, based on the caret vignette:

#Load Data
set.seed(42)
require(mlbench)
data(BostonHousing)
y <- BostonHousing[,14]
X <- BostonHousing[,1:13]

#Use the same CV-folds for each model
require(caret)
myControl <- trainControl(method='cv', number=10, index=createFolds(y, k=10))

#Fit models
model_rpart <- train(X, y, method='rpart', trControl=myControl)
model_ctree <- train(X, y, method='ctree', trControl=myControl)
model_rf <- train(X, y, method='rf', trControl=myControl)

#Plot
resamples <- resamples(list(
  rpart=model_rpart,
  ctree=model_ctree,
  model_rf=model_rf
  ))
dotplot(resamples, metric='RMSE')

In this example, the random forest has a lower cross-validated RMSE, so I would select that model.