How to find the optimal threshold for the Weighted f1 score in a binary classification problem I know how to find the optimal threshold for the standard f1 score but do not know how to do so for the weighted f1 score with the sklearn library. Sklearn provides a way to compute the weighted f1-score by passing average = 'weighted'. But it is unclear to me how I can retrieve a list of weighted f1-scores as the probability threshold of my true class prediction varies.
When dealing with the standard f1-score computation, the process is straightforward as I do not need to use the sklearn function to actually compute the f1-score.
For example I do the following:
precision, recall, thresholds = precision_recall_curve(y_true, y_score)
f1_scores = 2*recall*precision/(recall+precision)
print('Best threshold: ', thresholds[np.argmax(f1_scores)])
print('Best F1-Score: ', np.max(f1_scores))

I suppose the root of my problem is that I do not understand exactly how the "weighted f1-score" is calculated otherwise I could compute it manually as I did with the standard f1-score.
How can I find the optimal weighted f1 score?
 A: The weighted average of any array a is just
weight_avg = sum(a * weights) / sum(weights) but numpy average function accept weight as input.
You can compute directly the weighted_f1_scores using the the weights given by the number of True elements of each of the classes in y_true which is usually called support.
This can be obtained by just summing by rows the typical confusion matrix (I am referring to the the confusion matrix layout where the true labels are located in the rows and the predictions in the columns).
So assuming that besides the y_score you also have a y_pred with the default threshold (this is needed by confusion_matrix from sklearn.metrics, it does not work with scores)
Extending your code:
from sklearn.metrics import confusion_matrix
precision, recall, thresholds = precision_recall_curve(y_true, y_score)
f1_scores = 2*recall*precision/(recall+precision)
weights = confusion_matrix(y_true, y_pred).sum(axis=1)
weighted_f1_scores = np.average(f1_scores, weights=weights)
print('Best threshold: ', thresholds[np.argmax(weighted_f1_scores)])
print('Best F1-Score: ', np.max(weighted_f1_scores))


Actually sklearn is doing this under the hood, just using the np.average(f1_score, weights=weights) where weights = true_sum. true_sum is just the number of the cases for each of the clases wich it computes using the multilabel_confusion_matrix but you also can do it with the simpler confusion_matrix.  To see it you can see the code in sklearn.metrics._classification.py.
Computing the true_sum (lines 1464 to 1471):
    # Calculate tp_sum, pred_sum, true_sum ###
    samplewise = average == 'samples'
    MCM = multilabel_confusion_matrix(y_true, y_pred,
                                      sample_weight=sample_weight,
                                      labels=labels, samplewise=samplewise)
    tp_sum = MCM[:, 1, 1]
    pred_sum = tp_sum + MCM[:, 0, 1]
    true_sum = tp_sum + MCM[:, 1, 0]

Now applying the weighted average pre-processing (lines 1507 to 1526) which is basically assigning unless there are 0 cases:
    # Average the results
    if average == 'weighted':
        weights = true_sum
        if weights.sum() == 0:
            zero_division_value = np.float64(1.0)
     ....

And finally computing the weighted average (lines 1533 to 1540):
    if average is not None:
        assert average != 'binary' or len(precision) == 1
        precision = np.average(precision, weights=weights)
        recall = np.average(recall, weights=weights)
        f_score = np.average(f_score, weights=weights)
        true_sum = None  # return no support

   return precision, recall, f_score, true_sum

