Suppose I've built n number of different binary classification models. For what it's worth, they all use the exact same input and output data for training, are evaluated on the same test and validation sets.
Now, outside of looking at ROC-AUC, I've also looked at precision, recall, and F1 for a specific probability threshold. Currently, the threshold chosen depends on each model - this cutoff is determined by finding the probability that maximizes F1 for the predicted test-set probabilities for each model. As such, this threshold can vary (it only varies slightly in this case - between 0.40 and 0.45 for roughly 75 different model configurations).
My question: is it appropriate to evaluate each model's performance by looking at the F1-score for the one probability threshold determined by the strategy defined above?
For a more concrete example, here's a sample of the performance metrics and associated thresholds:
f1 threshold 0.75 0.40 0.71 0.41 0.74 0.44 0.77 0.45
Would it be appropriate/fair to take the fourth model (with an F1 of 0.77) as the final model?
I'm concerned with the chance the the predicted probability distributions could differ between the models, perhaps causing some problems with this model-selection strategy.