19
$\begingroup$

How does randomForest package estimate class probabilities when I use predict(model, data, type = "prob")?

I was using ranger for training random forests using the probability = T argument to predict probabilities. ranger says in documentation that it:

Grow a probability forest as in Malley et al. (2012).

I simulated some data and tried both packages and obtained very different results (see code below)

enter image description here

So I know that it uses a different technique (then ranger) to estimate probabilities. But which one?

simulate_data <- function(n){
  X <- data.frame(matrix(runif(n*10), ncol = 10))
  Y <- data.frame(Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %>%
                               pnorm(mean = 5)
                             ) %>% 
                    as.factor()

  ) 
  dplyr::bind_cols(X, Y)
}

treino <- simulate_data(10000)
teste <- simulate_data(10000)

library(ranger)
modelo_ranger <- ranger(Y ~., data = treino, 
                                num.trees = 100, 
                                mtry = floor(sqrt(10)), 
                                write.forest = T, 
                                min.node.size = 100, 
                                probability = T
                                )

modelo_randomForest <- randomForest(Y ~., data = treino,
                                    ntree = 100, 
                                    mtry = floor(sqrt(10)),
                                    nodesize = 100
                                    )

pred_ranger <- predict(modelo_ranger, teste)$predictions[,1]
pred_randomForest <- predict(modelo_randomForest, teste, type = "prob")[,2]
prob_real <- apply(teste[,1:10], 1, sum) %>% pnorm(mean = 5)

data.frame(prob_real, pred_ranger, pred_randomForest) %>%
  tidyr::gather(pacote, prob, -prob_real) %>%
  ggplot(aes(x = prob, y = prob_real)) + geom_point(size = 0.1) + facet_wrap(~pacote)
Improve this question
$\endgroup$
  • 1
    $\begingroup$ Just out of curiosity, what would be prob_real? $\endgroup$ – Firebug Jul 28 '16 at 20:56
  • 1
    $\begingroup$ The real response probability. As this is a simulation i have this for each observation $\endgroup$ – Daniel Falbel Jul 28 '16 at 20:58
17
$\begingroup$

It's just the proportion of votes of the trees in the ensemble.

library(randomForest)

rf = randomForest(Species~., data = iris, norm.votes = TRUE, proximity = TRUE)
p1 = predict(rf, iris, type = "prob")
p2 = predict(rf, iris, type = "vote", norm.votes = TRUE)

identical(p1,p2)
#[1] TRUE

Alternatively, if you multiply your probabilities by ntree, you get the same result, but now in counts instead of proportions.

p1 = predict(rf, iris, type = "prob")
p2 = predict(rf, iris, type = "vote", norm.votes = FALSE)

identical(500*p1,p2)
#[1] TRUE
Improve this answer
$\endgroup$
  • 2
    $\begingroup$ Thanks! Do you have any idea why proportion of votes is better then probability forests? Or you think this happens just for this problem? See this link (in portuguese) $\endgroup$ – Daniel Falbel Jul 28 '16 at 21:08
  • 2
    $\begingroup$ @DanielFalbel While I'm quite familiarized with randomForest I'm not much knowledgeable about ranger (in fact, I never used it), so I wouldn't be able to answer, I'm sorry. But it's an interesting question, perhaps you could make another question about how both strategies are different. $\endgroup$ – Firebug Jul 28 '16 at 21:14
7
$\begingroup$

The Malley (2012) is available here: http://dx.doi.org/10.3414%2FME00-01-0052. A full reference is in the references part in the ranger documentation.

In short, each tree predicts class probabilities and these probabilities are averaged for the forest prediction. For two classes, this is equivalent to a regression forest on a 0-1 coded response.

In contrast, in randomForest with type="prob" each tree predicts a class and probabilities are calculated from these classes.

In the example here I tried to use the uniform distribution instead of the normal distribution to generate the probabilities, and here the other approach seems to perform better. I wonder if these probabilities are really the truth?

By the way, the same results as in the randomForest example above can be achieved with ranger by using classification and manual probability computation (use predict.all=TRUE in prediction).

Improve this answer
$\endgroup$
  • $\begingroup$ you can see that those are the probabilities of response in the simulation code. Look at: Y = rbinom(n, size = 1, prob = apply(X, 1, sum) %>% pnorm(mean = 5)). That's how Y is generated, summing X1, X2, ..., X10 and then getting the quantile of the normal distribution with mean = 5 the sum represents. Do you think this makes sense? $\endgroup$ – Daniel Falbel Jul 30 '16 at 10:37
-1
$\begingroup$

If you want Out-Of-Bag probability estimates, you only can do it in randomForest package in R using model$votes. The other probability estimates are not OOB.

Improve this answer
$\endgroup$
  • $\begingroup$ what is OOB probability estimate? $\endgroup$ – user158565 Jan 11 '19 at 3:30
  • $\begingroup$ It is out of bag probability estimate. In a random forest, one way they estimate the probability associated with each class is they calculate the proportion of the trees that voted for each class. The OOB estimate would do the same but only count those trees' votes that the instance was not used in their training (aka the instance was not in-bag) $\endgroup$ – Max Jan 12 '19 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.