In binary classification, what is the optimum probability threshold to predict binary outcomes (0/1) on unseen data without knowing the actual outcome?

Let's assume that a random forest model has been trained on a training dataset using n-fold cross validation and the classification probability threshold is set to the value maximizing the F1 score.

Thus, we have a training probability threshold and a cross validation probability threshold.

Which one is to be used for predicting the class (0/1) of the unseen data when applying the model?


1 Answer 1


I have two remarks, which do not answer your question:

1- when you train a RF model you do not need to perform cross validation: https://datascience.stackexchange.com/questions/6510/does-modeling-with-random-forests-requre-cross-validation

2- which threshold you choose has nothing to do with statistics, it is a decision you need to take depending on the problem at hand (this is a really nice answer): Classification probability threshold

So I guess this second point helps me to rephrase your question: assuming that I want to maximize the F1 score also in the unseen data (test set), which threshold should I use, the one maximising the F1 score in the cross validation or in the training?

I think the one in the training. Because cross validation just helps you choose/validate the model. Also I do not expect the two thresholds should differ much in absence of overfitting.

  • $\begingroup$ Agree on the methodology, however this does not answer my question. I want to use cross-validation, it gives more robust results. Essentially my question is, after I have done all of the above, and get a new batch of data to score, what is the "best" threshold to apply in order to tranform scores into predictions? Could be the one from the independent test set...however, sometimes the test set is small and may not capture seasonality in data. Thanks for taking the time though. $\endgroup$
    – mincorp
    May 9, 2018 at 16:01
  • $\begingroup$ You are right, sorry, I totally misunderstood the question. I edited my answer accordingly. $\endgroup$
    – fabiob
    May 9, 2018 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.