I have a classification problem where the main aim is maximising classification accuracy. I am using a random forest, and would like to also use the variable importance in my analysis.
For this reason, I thought it best to use
cforest from the
party package in R since it has been shown Strobl et al. 2007 that it gives unbiased variable importance.
However, I found that it is giving +-20% less accuracy when compared to the
randomForest package. To my understanding there is not much between the two implementations so I did not expect such a large difference. I also read that
cforest is more aligned to ordinal dependent variables, so I was actually expecting an increase in accuracy!
Can someone familiar with random forests please tell me if this is normal behaviour or if there is something else I am missing?
df <- read.csv(url("https://raw.githubusercontent.com/allandf/r-funcs/master/data/testdata.csv")) df$rating <- as.ordered(df$rating) require(caret) #Split data according to 80% training, 20% test flds <- createFolds(df[,"rating"], k = 4, list = TRUE, returnTrain = FALSE) test_data <- df[flds$Fold1,] train_data <- df[flds$Fold2,] train_data <- rbind(train_data,df[flds$Fold3,]) train_data <- rbind(train_data,df[flds$Fold4,]) require(party) set.seed(1234) my_cforest_control <- cforest_control(teststat = "quad", testtype = "Univ", mincriterion = 0, ntree = 500, mtry = 5, replace = FALSE) my_cforest <- cforest(rating ~ ., data = train_data, controls = my_cforest_control) cforest_pred <- predict(object = my_cforest, newdata = test_data[,-9]) confusionMatrix(cforest_pred, test_data[,9]) require(randomForest) set.seed(1234) my_randomForest <- randomForest(rating ~ ., data = train_data, importance = TRUE, ntree = 500, mtry = 5, replace = FALSE) randomforest_pred <- predict(object = my_randomForest, newdata = test_data[,-9]) confusionMatrix(randomforest_pred, test_data[,9])
I also tried first learning the parameter
mtry using 10 fold cross validation, but it did not make a difference. I also tried all variations of
testtype mentioned in the
EDIT: I thought it might be a numerical issue, so I tried scaling the dataset first, unfortunately this didnt make a difference.