0
$\begingroup$

I've trained a simple NN to perform binary classification with goal of maximizing area under ROC curve. Right now AUC is around 0.85. Out of curiosity, I checked which thresholds are best in terms of maximizing f1_score. It turned out that optimal thresholds are around 0.08 < 0.1, corresponding to fpr = 0.22 and tpr = 0.72 and f1_score = 0.75. Note that training and evaluation datasets are inbalanced with ~90% of negative and 10% of positive samples.

I am wondering what does it mean about the data or the model if f1-optimal threshold is so low and how can I use that knowledge to improve my model. My initial guess was that low threshold is result of unbalanced classes - because only 10% of samples is positive, it makes sense to be very sensitive and classify something as positive even with low certainty, but then I realized it should be the opposite.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ I think this is exactly what I discuss in my question here. $\endgroup$
    – Dave
    Mar 20 at 15:52
  • 1
    $\begingroup$ why is maximising the F1 score appropriate for your application? $\endgroup$
    – Dikran Marsupial
    Mar 20 at 15:55
  • 1
    $\begingroup$ @DikranMarsupial It isn't, I just checked out of curiosity. I tried also minimizing distance to (0,1) corner and got similar threshold. In general, the task is from old Kaggle competiton and the only metric they used to evaluate models were AUC ROC. $\endgroup$
    – Brzoskwinia
    Mar 20 at 16:01
  • $\begingroup$ What kind of FPR, TPR, and $F_1$ score do you get when you predict every observation to be the majority category? $\endgroup$
    – Dave
    Mar 20 at 16:03
  • 1
    $\begingroup$ @Brzoskwinia thanks for clarifying. I think a lot of the misinformation about imbalanced datasets is caused by not having a well thought out performance metric for the application. I would experiment with coming up with some misclassification costs for the task and see how to adjust the threshold to minimise the expected loss. IMHO if you don't know the misclassification cost, you are better off avoiding performance metrics that depend on the threshold as the threshold is really a statement about the misclassification costs. Just leaving it as AUROC or a proper scoring rule is better. $\endgroup$
    – Dikran Marsupial
    Mar 20 at 16:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.