Sudden drop to zero for precision recall curve

Question

I am training a neural network classifier with 250k training samples and 54k validation samples. The output activation is sigmoid. I noticed a sudden drop in the precision for the very top probability scores. The graph looks as below. Any advice about a possible cause?

Answer 1

The Precision is likely tanking because when we use a very high threshold, we end up with no True Positives, even though we might have some False Positives (i.e. negative examples we incorrectly assign a very high probability of being positive).

This drop is not too uncommon, it might signify mislabelling, edge cases or simply a slightly underfitted model. For starters, I would recommend plotting a calibration plot (e.g. see for example here for a sklearn functionality) such that you can assess the coherence/monotonicity of your predicted scores. If it is problematic (e.g. saw-tooth shaped), work on getting a more well-calibrated classifier; Calibration: the Achilles heel of predictive analytics (2019) is a great resource if you want to read a bit more on this. In addition to that, do some explainability analysis on your classifier too. Check which features are the ones affecting predictions the most and then investigate if the mislabeled examples have extreme values, this can highlight mislabeling or edge cases that might otherwise perplex us. A nice accessible blog post on this is "Machine learning interpretability with feature attribution". A real-life example of this would be the following: we identify that being older, male and overweight are significant risk factors for a particular symptom $C$ based on our model $M$, but the handful of morbidly obese older male patients in our test sample, do not have the symptom $C$. Our model's precision for a very high threshold will be likely $0$ despite $M$ being a potentially reasonable model overall.