In binary classification, what is the optimum probability threshold to predict binary outcomes (0/1) on unseen data without knowing the actual outcome?
Let's assume that a random forest model has been trained on a training dataset using n-fold cross validation and the classification probability threshold is set to the value maximizing the F1 score.
Thus, we have a training probability threshold and a cross validation probability threshold.
Which one is to be used for predicting the class (0/1) of the unseen data when applying the model?