What is the F1 Score for my prediction when all values are negative?

Question

I have built a model that gives me classification of some cases

here is a comparison between Actual and Prediction

Prediction   Actual
------------------------
    0          0
    0          0
    0          0
    0          0
    0          0
    0          0
    0          0
    0          0
    0          0

They all match and they all negative

when I try to get F1 Score for this case I get divided by zero

which makes sense as the Precision is TP / TP+FP (which all are zeros)

so in my case what would the F1 score be?

Answer 1

In your case, where all predictions and actual values are zero, calculating the F1 score results in division by zero because both precision and recall are undefined. Here’s the breakdown:

1. True Positives (TP): 0
2. False Positives (FP): 0
3. False Negatives (FN): 0

Calculations:

• Precision is undefined since TP and FP are both 0.
• Recall is undefined since TP and FN are both 0.
• Consequently, the F1 score is also undefined.

Conclusion:

In this scenario, it is common to define the F1 score as 0, as the model has not identified any positive instances correctly. Many libraries may return either nan or 0 for the F1 score in such cases. It’s good practice to handle this specifically in your evaluation metrics by returning 0 when both precision and recall are undefined.

---

Another example to go further:

In this example, we will consider a scenario where we add one to each category of the classification results. Specifically, we will assume:

• True Positives (TP): 0 → becomes 1 after the adjustment.
• False Positives (FP): 0 → becomes 1 after the adjustment.
• False Negatives (FN): 0 → becomes 1 after the adjustment.
• True Negatives (TN): We will also assume there are 1,000 true negatives.

This will allow us to evaluate the impact on the F1 score and other metrics under these conditions.

Calculating the Metrics:

Precision:

• Precision measures how many of the predicted positive cases were actually positive. It's calculated as:

Precision = TP / (TP + FP) = 1 / (1 + 1) = 0.5

This means that half of the predicted positive cases were correct.

Recall:

• Recall measures how many actual positive cases were identified by the model. It’s calculated as:

Recall = TP / (TP + FN) = 1 / (1 + 1) = 0.5

This indicates that the model successfully identified half of the actual positive cases.

F1 Score:

• The F1 score is the harmonic mean of precision and recall, providing a single score that balances both metrics. It can be calculated as:

F1 = 2 * (Precision * Recall) / (Precision + Recall) = 2 * (0.5 * 0.5) / (0.5 + 0.5) = 0.5

So, the F1 score in this case is 0.5.

The Role of True Negatives:

While you have 1,000 true negatives in this scenario, it’s important to note that true negatives do not impact the F1 score directly. The F1 score is primarily concerned with how well the model identifies the positive class (i.e., true positives, false positives, and false negatives). True negatives contribute to overall accuracy but not to the F1 score.

Conclusion:

In summary, even with a substantial number of true negatives, the adjustments made to the counts (adding +1 to TP, FP, and FN) lead to an F1 score of 0.5. This score reflects the balance between precision and recall for the positive class. While true negatives are essential for understanding model performance overall, they don’t influence the F1 score, which focuses on the model’s ability to correctly identify positive cases.

Answer 2

In this case, you cannot calculate F1-score. You cannot also calculate precision and recall score. Use accuracy instead of them.

Answer 3

I think you already answered your question: Divided by Zero. F1-score is calculated from Precision and Recall. Your example only contains negative case. So you can't calculate positive related items(Precision). In human words: You don't have enough data to evaluate this model.