I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a improper scoring rule for model optimisation.
However, when I perform a hyperparameter optimisation
RandomizedSearchCV and set scoring to
"f1_weighted" the performance metrics on the hold-out test set is better than the with the scoring set to
'neg_log_loss'. However, I am not sure whether it is justified to use
f1_weighted in this case?
Furthermore, when I plot the f1_weighted during training, it is very much underfitting (train = 0.64 and test is 0.75) when compared to the hold-out test performance, while this is not the case for the log-loss score (train = 0.6 and test = 0.58). Does this indicate that the log loss approach is the way to go?