Use F1 or maximum F1 for model comparisons? I am comparing a ML classifier to a bunch of other benchmark F1 classifiers by F1 scores. By AUPRC, my classifier does worse than other benchmark methods. When I compared F1 score, however, I got a curious result that my classifier does better than the other methods. After looking at the precision recall curve, I realized that indeed my model was performing better in precision/recall at the probability threshold where the F1 score was being evaluated by default(classifier probability > 0.5), but performed worse at other thresholds. 
When I report these results, should I be looking at maximum possible F1 score when comparing between models? Please note that I am working with an imbalanced dataset, and the precision/recall tradeoff matters for my use case (hence the F1 as the performance metric).
 A: Even if you are looking for rare events in an unbalanced data set such that a focus on precision and recall is warranted and true negatives aren't of interest, you should be wary of using the $F_1$ score as your way to compare models.
The $F_1$ score is a specific weighting (harmonic mean) of precision and recall predicated on equal importance of precision and recall. If you really and truly consider them equally important, then fine. More generally, if "recall is considered $\beta$ times as important as precision" then you should use the $F_\beta$ score instead:
$$F_\beta = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}. $$
This reduces to $F_1$ for equal weighting of precision and recall. So even in this circumstance it still is important to consider the (often hidden) tradeoffs in choosing a performance measure.
Once you have taken into account the relative importance of precision and recall for your application with some $F_\beta$ score, I agree with @usεr11852 that you should then find (in some careful, generalizable way) the predicted-probability cutoff that maximizes the score. So for comparing among models it would seem best to use the maximum score achievable. If you choose $\beta =1$, then that's the maximum $F_1$ score.
