Classification threshold selection for predictions on unseen data

Question

In binary classification, what is the optimum probability threshold to predict binary outcomes (0/1) on unseen data without knowing the actual outcome?

Let's assume that a random forest model has been trained on a training dataset using n-fold cross validation and the classification probability threshold is set to the value maximizing the F1 score.

Thus, we have a training probability threshold and a cross validation probability threshold.

Which one is to be used for predicting the class (0/1) of the unseen data when applying the model?

Answer 1

I have two remarks, which do not answer your question:

1- when you train a RF model you do not need to perform cross validation: https://datascience.stackexchange.com/questions/6510/does-modeling-with-random-forests-requre-cross-validation

2- which threshold you choose has nothing to do with statistics, it is a decision you need to take depending on the problem at hand (this is a really nice answer): Classification probability threshold

So I guess this second point helps me to rephrase your question: assuming that I want to maximize the F1 score also in the unseen data (test set), which threshold should I use, the one maximising the F1 score in the cross validation or in the training?

I think the one in the training. Because cross validation just helps you choose/validate the model. Also I do not expect the two thresholds should differ much in absence of overfitting.