I'm testing the randomForest package in R and observing something really strange. Let's first look at the R code below:
n <- 1000 set.seed(12345) X <- matrix(rbinom(n, 1, 0.5), n, 1) X <- 2 * X - 1 e <- 1 / (1 + exp(X)) W <- rbinom(n, 1, e) dat <- cbind(W, X) dat <- data.frame(dat) dat$W <- factor(dat$W) rf.classif <- randomForest(W ~ ., data = dat, ntree = 1000) e.prob <- predict(rf.classif, newdata = dat, type = "prob")[, 2] e.prob <- as.numeric(e.prob) e.prob[1:10]
I use one single binary covariate/feature $X$ to predict $W$. As we can see, $W$ is Bernoulli distributed with probability $e$, which has value of either $0.2689414$ or $0.7310586$. I use randomForest function in R with $1000$ trees (well, since we have only one binary covariate $1$ tree should give the same result).
Anyway, the predicted probability for $W = 1$ ($e.prob$) returned by randomForest is very strange: it is either $0$ or $1$ instead of some value in between. The first $10$ values of $e.prob$ look like:
 0 0 0 0 1 1 1 0 0 0
I'm trying multiple ways to resolve this problem but still getting the same result. As a side note, running a simple logistic regression would give a very good predicted probability for $W = 1$.
If Random Forest can overfit, then why cannot it give good predicted probability in this simple setting?
Any ideas/suggestions/insights would be very much appreciated.