Random Forests performs far less better than Naive Bayes

Question

In one of my Machine Learning courses I have to find the best predictor for this dataset and its binary target "Caesarian".

First of all, I tried to improve the datas : there are few features. I did One-Hot-Encoding on :

• Delivery Time
• Blood Pressure
• Heart Problem (already done in the database as binary)

I normalized my datas for the Age and Delivery Number columns.

Then I did k-fold with Random Forests, Naive Bayes, GLM and some other, I tried several k for the k-fold, $k\in \{1,2,3,6,8,16\}$, and I was very surprise by the very poor performances on prediction for the Naive bayes (more than 30% error on average for the k-fold, no matter what k), and most of all, I was extremely surprise that RandomForest performs a 40% mistake on average, no matter what $k$ and with different values for the number of trees, from tens to thousands, and also different values for the number of variables ramdomly sampled as candidates at each split.

I tried Leave-One-Out to compute the error, and it's the same, the Naive Bayes model's quite bad, and still surprisingly, the Random Forest's worst than Naive Bayes.

Is there something I missed out ? I can understand that Naive Bayes can have a bad classification performance as it really depends on the datas and it's a really weak classifier, but what about RandomForest ?

So two questions now :

1. Random Forest aren't suppose to perform better ?
2. What can I do to improve them or at least understand why it's so bad ?

Here are the boxplots of the MSE I got for a 6-fold :

Note : I didn't tried to optimize Bagging yet

Answer 1

It depends on the "true" data generation process. If the model does not match this process, then it will not perform as well as a model (perhaps a simpler one) that does match it. So a GLM typically performs worse than a random forest, but not when the true data generation process is exactly like a linear regression.

For example: I simulate data below (using R) from a logistic regression model. A GLM has a higher accuracy (85%) than a random forest (79%) on the test set.

In general, models perform better when their form and assumptions match what truly generated the data. Our models are never perfect, but they perform well when they approximate the process close enough.

library(randomForest)
inv_logit <- function(x) exp(x) / (1 + exp(x))
set.seed(1839)
n <- 1000
x1 <- rnorm(n)
x2 <- runif(n)
x3 <- rpois(n, 4.5)
y_prob <- inv_logit(
  -.2 + .3 * x1 + .1 * x2 + -.2 * x3 + # main effects
    x1 * x2 + 1.4 * x2 * x3 + .1 * x1 * x2 * x3 # interactions 
)
y <- rbinom(n, 1, y_prob)

cases <- sample(seq_len(n), 700)
dat <- data.frame(x1, x2, x3, y)
train <- dat[cases, ]
test <- dat[-cases, ]

mod1 <- glm(y ~ x1 * x2 * x3, binomial, train)
pred1 <- predict(mod1, test, type = "response") > .5
sum(diag(table(test$y, pred1))) / sum(table(test$y, pred1))

mod2 <- randomForest(factor(y) ~ ., train)
pred2 <- predict(mod2, test)
sum(diag(table(test$y, pred2))) / sum(table(test$y, pred2))

Answer 2

As Mark notes, how accurate you can be depends on the data.

For example, if a stock market index is really a random walk, you aren't going to be able to predict it well.

I took this data set and let it run for 30 minutes in an automated program that searches through a bunch of models. Even after that time, the crossvalidation accuracy was 72.5% (27.5% incorrect). That's not much better than Naive Bayes.

Because running the data through a fully automated procedure is not likely to be what you Machine Learning instructor wants, I'm going to skip the details.

In general, I've found Random Forests to work better than Naive Bayes, but there are certainly exceptions. Methods that never work better tend to disappear; methods that are still used tend to work well in some set of circumstances.

Answer 3

In my opinion, a potentially more important reason as to why random forests are doing quite poorly here is mostly due to the size of your dataset.

Your dataset is comprised of a grand total of 80 observations with five features which is incredibly small for fitting a random forest. Consider how a classification tree generates its predictions, the base learner for a random forest ensemble. The final predicted probability of being in a specific class will just be the proportion of observations (in a single terminal node of the tree) that belong to this class to the total number of observations in that same terminal node. A random forest fits many of these decision trees on bootstrap samples of your data and then takes an average of all of the trees to get the final prediction. Thus, with only 80 observations your fitted probabilities of belonging to one class or the other will be based off very few observations which will more than likely lead to large variances in your predictions (i.e. severe overfitting). Thus, it is really not surprising to see that simpler models work better here because a random forest in general requires a large amount of observations to work well. This also holds true for models such as neural networks, gradient boosting, regularized greedy forests, and other very (popular) methods that many regard as "high performing" algorithms (as of 2018, anyway).

In general, I have found that the less data you have, the less complex your models can be. Ensemble based methods and models that fit a large amount of parameters do not fare well on small amounts of data because there simply isn't enough information to estimate very complicated relationships within the data; which is in general why most people use these more complex algorithms in the first place.