Selecting best classification probability threshold with ROC/AUC doesn't necessarily improve F1 score I read that probability based binary classifiers have 0.5 as default probability threshold for getting hard 0/1 labels (in scikit-learn for example) but this could be fine-tuned with methods like receiver operator characteristics. I have run experiments in multiple times (best candidate is defined by Youden's J statistics) like the one illustrated below, however, F1-score didn't necessarily improved (sometimes they did, sometimes didn't) if I changed the original 0.5 threshold to the one found with ROC+J. What is the reason for the divergence? ROC is not necessary for fine-tuning for F1?
My experiment (in Python with Sklearn):
X, y = make_classification(100)
model = LogisticRegression()
model.fit(X, y)
y_pred = model.predict(X)
y_pred_proba = model.predict_proba(X)
y_pred_proba = y_pred_proba[:,1]

fpr, tpr, thresholds = roc_curve(y, y_pred_proba)

best_index = np.argmax(tpr - fpr)
best_threshold = thresholds[best_index]

print(f1_score(y, y_pred)) # This is with the original 0.5 threshold
print(f1_score(y, np.where(y_pred_proba > best_threshold, 1, 0))) # This is with my found threshold, sometimes with lower F1

 A: You have two metrics, the F1-score on the one hand and Youden's J on the other hand. Those two metrics measure different aspects of the model which are not functions of each other, so you cannot deduce Youden's J from the F1-score or the other way around. Thus one should not expect the improvement of one to necessarily improve the other.

Let $TP, TN, FP, FN$, denote true positives, true negatives, false positives, and false negatives, resp.
F1 is the (weighted) harmonic mean of the precision ($\frac{TP}{TP + FP}$) and the recall ($\frac{TP}{TP+FN}$). Furthermore, Youden's J uses also recall, but the second variable it uses is the false positive rate FPR ($\frac{FP}{FP+TN}$): J = recall - FPR. Now, e.g. if you increase both $FP$ and $TN$ proportionally, such that the FPR stays constant, Youden's J will stay constant but the F1 score will suffer, since, while the recall stays constant, the precision will decrease. In the same way, by decreasing both $FP$ and $TN$ proportionally, you can increase F1.
Note, that F1 is independent of the size of $TN$.
