Kind of a broad question here. But is it okay/possible in R to use a random forest for regression when the response variable is a binary outcome? Essentially what I'm looking for is a probability of something happening. Below is the code I've been using it run it and the warning message. Or am I better off using this in classification mode. My end goal is to predict the probability of something happening, not necessarily predict what "classification" it will be, 1 or 0.

m1RF <- randomForest(EOI_140 ~ .,
                     data = dfTRN,
                     importance = TRUE,
                     ntree = 2000)
Warning message:
In randomForest.default(m, y, ...) :
  The response has five or fewer unique values.  Are you sure you want to do regression?

I apologize if this question is in the wrong place.

  • $\begingroup$ @ArtemSokolov: agreed it's a duplicate, although "Can RF regression be used for binary response var?" is more fundamental than that wording, and they don't share any common words, so non-obvious :) $\endgroup$
    – smci
    Commented Feb 5, 2018 at 22:53
  • $\begingroup$ @ArtemSokolov: oh and also, just because RF can predict class probabilities, we might well want to modify that raw number, depending on the evaluation function (we might take log1p, sigmoid, square, square-root, etc.) $\endgroup$
    – smci
    Commented Feb 5, 2018 at 22:56

2 Answers 2


Try this with "prob" in predict(). It is done on mtcars predict vs column

mtcars$vs <- as.factor(mtcars$vs)
classifier <- randomForest( formula = vs~hp+drat, data=mtcars)
predict(classifier, type="prob")

I dont split to train and test sample but in reality you have to do this :)

You have to set factor to response variable in other case it will try to do Random Forest Regression :)


Let's start with something simpler, a decision tree, since random forests are forests of independent decision trees that are averaged. Decision tree is build by splitting your data into subsets conditionally on the features used. The splits are done by choosing the features, one at a time, and then choosing a split based on the values of the feature. In both cases we make our choices based on some loss function that is minimized. In regression case we minimize the variance, what is equivalent to minimizing squared loss. In classification case, we use entropy, or Gini impurity as a criterion. In the end your data gets packed into a number of subgroups and to make predictions, in classification case you predict the most frequent value within the subgroup, and in regression case you predict the mean of the subgroup. Obviously, if you calculate the mean of the binary values, you'd get the fraction, i.e. empirical probability. So basically in both cases you can calculate probabilities the same way, this problem reduces only to the criteria that is used for building the tree: mean squared error vs entropy (or Gini impurity). If you choose mean squared error, then you'd be minimizing the same loss as linear regression, while choosing entropy, leads to minimizing the same loss function as logistic regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.