I have a classification problem where the main aim is maximising classification accuracy. I am using a random forest, and would like to also use the variable importance in my analysis.

For this reason, I thought it best to use cforest from the party package in R since it has been shown Strobl et al. 2007 that it gives unbiased variable importance.

However, I found that it is giving +-20% less accuracy when compared to the randomForest package. To my understanding there is not much between the two implementations so I did not expect such a large difference. I also read that cforest is more aligned to ordinal dependent variables, so I was actually expecting an increase in accuracy!

Can someone familiar with random forests please tell me if this is normal behaviour or if there is something else I am missing?

df <- read.csv(url("https://raw.githubusercontent.com/allandf/r-funcs/master/data/testdata.csv"))
df$rating <- as.ordered(df$rating)

#Split data according to 80% training, 20% test
flds <- createFolds(df[,"rating"], k = 4, list = TRUE, returnTrain = FALSE)
test_data <- df[flds$Fold1,]
train_data <- df[flds$Fold2,]
train_data <- rbind(train_data,df[flds$Fold3,])  
train_data <- rbind(train_data,df[flds$Fold4,])  

my_cforest_control <- cforest_control(teststat = "quad",
    testtype = "Univ", mincriterion = 0, ntree = 500, mtry = 5,
    replace = FALSE)

my_cforest <- cforest(rating ~ ., data = train_data,
                      controls = my_cforest_control)
cforest_pred <- predict(object = my_cforest, newdata = test_data[,-9]) 
confusionMatrix(cforest_pred, test_data[,9])

my_randomForest <- randomForest(rating ~ ., data = train_data,
                                importance = TRUE, ntree = 500,
                                mtry = 5, replace = FALSE)

randomforest_pred <- predict(object = my_randomForest, newdata = test_data[,-9])
confusionMatrix(randomforest_pred, test_data[,9]) 

I also tried first learning the parameter mtry using 10 fold cross validation, but it did not make a difference. I also tried all variations of teststat and testtype mentioned in the party documentation.

EDIT: I thought it might be a numerical issue, so I tried scaling the dataset first, unfortunately this didnt make a difference.

  • 1
    $\begingroup$ I had similar experience, seems Leo's random forest is really good in many data sets. other models, although the author may argue better, are not that good in may real world data. $\endgroup$ – hxd1011 Mar 9 '18 at 15:11

The "problem" is that your response is an ordered categorical variable at many levels, and neither randomForest nor cforest were designed to deal with this case. cforest uses linear scores (which might or might not be appropriate). While tree induction and forest aggregation might work, variable importances (and model comparison) will break because misclassification error and its friends are telling the wrong story. These measures ignore the ordering and you want to take this information into account.

What can be done? In a nutshell, one can start with a proportional odds model and then use model-based trees (and the corresponding forests) to improve upon this initial unconditional model. The key point is that the proportional odds model gives you a likelihood and this is an appropriate way to assess risk in this situation. Here is some (still experimental and relatively slow) code for your dataset:

df <- read.csv(url("https://raw.githubusercontent.com/allandf/r-funcs/master/data/testdata.csv"))
df$rating <- as.ordered(df$rating)

#Split data according to 80% training, 20% test
flds <- createFolds(df[,"rating"], k = 4, list = TRUE, returnTrain = FALSE)

ll <- matrix(0, nrow = length(flds), ncol = 3)
colnames(ll) <- c("Polr", "Tree", "Forest")
for (i in 1:length(flds)) {
    fd <- flds[[i]]
    win <- as.numeric(!(1:NROW(df) %in% fd))

    ### fit unconditional proportional odds model
    ### see vignette("tram", package = "tram") and
    ### vignette("mlt", package = "mlt.docreg")
    m <- Polr(rating ~ 1, data = df, weights = win)
    ### out of sample log-likelihood
    ll[i, "Polr"] <- logLik(m, w = 1 - win)

    ### partition unconditional model the mob-way,
    ### see https://arxiv.org/abs/1701.02110 -- describes continuous case only,
    ### but DOI: 10.1111/sjos.12291 explains the ordered categorical case
    mt <- trafotree(m, formula = rating ~ ., data = df, weights = win, 
                    control = ctree_control(alpha = .0001, minsplit = 100))
    ### out of sample log-likelihood
    ll[i, "Tree"] <- logLik(mt, w = 1 - win)

    ### corresponding forest
    mf <- traforest(m, formula = rating ~ ., data = df, 
                    ntree = 50, weights = win, mtry = 5, trace = TRUE, 
                    cores = 3)
    ### out of sample log-likelihood
    ll[i, "Forest"] <- logLik(mf, w = 1 - win)

### (conditional) variable importance
mf <- traforest(m, formula = rating ~ ., data = df, 
                ntree = 50, mtry = 5, trace = TRUE, cores = 3)
### likelihood-based variable importance
### likelihood-based conditional variable importance
varimp(mf, conditional = TRUE)
### predictions are conditional distribution functions
predict(mf, newdata = df[1:3,], type = "distribution")
### or densities
predict(mf, newdata = df[1:3,], type = "density")

One can speed-up things quite a bit by using a smooth basis function for your ordered response and some other tricks, but computing time is cheap these days and, especially in cases like this, worth the buck!


Torsten ps. Please contact me by email if you have questions. I'm not following discussions on other channels.


You can also use ordinal forests. In this case it is not better than the standard random forest, at least for the test dataset. For a better evaluation you should use cross-validation (for example via mlr package) instead of a simple test dataset, which randomly can provide better results for the standard random forest.

Example code for ordinal forest:

ordforest <- ordfor(depvar="rating", data=train_data, nsets=1000, ntreeperdiv=100, 
  ntreefinal=5000, perffunction = "equal")

ordforest_pred = predict(ordforest, newdata = test_data[,-9])
confusionMatrix(ordforest_pred$ypred, test_data[,9]) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.