Scoring rules (log-loss vs. F1-weighted) and RandomizedSearchCV

I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a improper scoring rule for model optimisation.

However, when I perform a hyperparameter optimisation RandomizedSearchCV and set scoring to "f1_weighted" the performance metrics on the hold-out test set is better than the with the scoring set to 'neg_log_loss'. However, I am not sure whether it is justified to use f1_weighted in this case?

Furthermore, when I plot the f1_weighted during training, it is very much underfitting (train = 0.64 and test is 0.75) when compared to the hold-out test performance, while this is not the case for the log-loss score (train = 0.6 and test = 0.58). Does this indicate that the log loss approach is the way to go?

"The way to go," as you will read on many threads on this site, is to (1) develop a well-calibrated probability model with a strictly proper scoring rule, for example log loss, and then (2) choose a probability cutoff (if needed) that represents your tradeoff between costs of false-negative and false-positive assignments. With a well calibrated model and costs scaled so that c is the cost of a false positive and 1-c the cost of a false negative, you minimize your cost by choosing a probability cutoff of c. See this page for details and links to further reading. If you think in terms of relative costs, things like F1 score will take care of themselves.