1
$\begingroup$

I read multiple posts about scoring rules during cross-validation and the fact that the log-loss score is a proper scoring rule and, correct me if I am wrong, any threshold based approach is a improper scoring rule for model optimisation.

However, when I perform a hyperparameter optimisation RandomizedSearchCV and set scoring to "f1_weighted" the performance metrics on the hold-out test set is better than the with the scoring set to 'neg_log_loss'. However, I am not sure whether it is justified to use f1_weighted in this case?

Furthermore, when I plot the f1_weighted during training, it is very much underfitting (train = 0.64 and test is 0.75) when compared to the hold-out test performance, while this is not the case for the log-loss score (train = 0.6 and test = 0.58). Does this indicate that the log loss approach is the way to go?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The log-loss and F1 scores really can't be compared. The log loss evaluates the full probability model. The F1 score suffers from (1) being based on an assumption of a probability cutoff (often a hidden assumption of p = 0.5) and (2) ignoring true negatives.

"The way to go," as you will read on many threads on this site, is to (1) develop a well-calibrated probability model with a strictly proper scoring rule, for example log loss, and then (2) choose a probability cutoff (if needed) that represents your tradeoff between costs of false-negative and false-positive assignments. With a well calibrated model and costs scaled so that c is the cost of a false positive and 1-c the cost of a false negative, you minimize your cost by choosing a probability cutoff of c. See this page for details and links to further reading. If you think in terms of relative costs, things like F1 score will take care of themselves.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your reply. I was not trying to compare log-loss and F1 directly. What I wanted to emphasise was that when I optimise my hyperparameters during cross-validation and use log-loss as scoring argument then I obtain a poorer performing model (when looking at the confusion matrix) when compared to using the F1 as scoring argument. So, let me rephrase just to see if I understand: 1. perform hyperparameter optimisation with log-loss. 2. Set a threshold c that will be used to then classify the data (let's say threshold of 0.6) to obtain the optimal balance between TP/TN and FN/FP $\endgroup$
    – JonnDough
    Commented Jan 30, 2021 at 15:44
  • 1
    $\begingroup$ @Oorschelp you understand correctly. My main point is that using F1 as a training criterion can be unreliable. Log-loss, however, is only one of many potential proper scoring rules for training. See links from this answer for other possibilities if you have particular relative costs in mind to start. With a well-calibrated model then set your threshold to represent relative costs. You might need to play around some if the model isn't well calibrated; see the answer I linked above in this comment. $\endgroup$
    – EdM
    Commented Jan 30, 2021 at 15:55
  • $\begingroup$ One last question: Are you by any chance familiar with SciKit learn in python? Because I don't think I can include the threshold in the RandomizedSearchCV from Python during training. Or would it be that 1. hyperparameter optimisation with log-loss 2. fit the trained optimised model 3. obtain the predict_proba for hold-out test set and then set a threshold? This seems to me that it would be prone to overfit? $\endgroup$
    – JonnDough
    Commented Jan 30, 2021 at 16:05
  • 2
    $\begingroup$ @Oorschelp sorry, don't use SciKit Learn. Your outline is OK. To evaluate overfitting, repeat the entire modeling process (all 3 outlined steps) on multiple bootstrapped samples of your data and evaluate performance on the full original data sample. That's the standard "optimism bootstrap," providing information on bias, overfitting, and non-parametric estimates of coefficient errors. See for example this page, this page and this page. $\endgroup$
    – EdM
    Commented Jan 30, 2021 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.