I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a improper scoring rule for model optimisation.

However, when I perform a hyperparameter optimisation RandomizedSearchCV and set scoring to "f1_weighted" the performance metrics on the hold-out test set is better than the with the scoring set to 'neg_log_loss'. However, I am not sure whether it is justified to use f1_weighted in this case?

I will show a few pictures below regarding the confusion matrices and the performance on the test set. Note, I picked the seemingly best performing model (for log_loss = XGBoost and F1_weighted = RF). Note, the F1 in the test overview tables are not the F1_weighted but rather the regular F1.

Furthermore, when I plot the f1_weighted during training, it is very much underfitting (train = 0.64 and test is 0.75) when compared to the hold-out test performance, while this is not the case for the log-loss score (train = 0.6 and test = 0.58). Does this indicate that the log loss approach is the way to go?

I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a improper scoring rule for model optimisation.

However, when I perform a hyperparameter optimisation RandomizedSearchCV and set scoring to "f1_weighted" the performance metrics on the hold-out test set is better than the with the scoring set to 'neg_log_loss'. However, I am not sure whether it is justified to use f1_weighted in this case?

Furthermore, when I plot the f1_weighted during training, it is very much underfitting (train = 0.64 and test is 0.75) when compared to the hold-out test performance, while this is not the case for the log-loss score (train = 0.6 and test = 0.58). Does this indicate that the log loss approach is the way to go?