# Measuring how well our service detects malfunctioning parts with different percentages?

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as 10%, 25%,etc. What I want to show is how well we predict malfunctioning parts.

For example, there are total of 100 system parts and 10 of them are malfunctioning. Our algorithm predicts 12 malfunctioning parts but in reality 8 of these predicted parts are malfunctioning. Hence, the precision of our model is 8/12=0.66 and the recall is 8/10=0.8. But, when the percentage of malfunctioning part is increased from 10 to 25, the precision will inevitably increase because we are dealing with another data partition for which we will have more true positives compared to false positives. So, precision here is not a good measure. We can still use recall for measuring the completeness of correct results. What I want to ask is should I use a metric like 𝑓𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 / (𝑓𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 + 𝑡𝑟𝑢𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠) to measure how many of irrelevant items are annotated as malfunctioning? Do we have a name for this metric?

• Do your algorithms produce numeric scoring? If so you can compute ROC-AUC and it is considered a good metric. Sep 8, 2021 at 11:32

Also, When I looked at https://en.wikipedia.org/wiki/Sensitivity_and_specificity, I thought you may be interested in the balanced accuracy metric among the listed measurements: $$BA = (TPR + TNR)/2$$. This is similar to ROC-AUC in that the trade-off between TPR and TNR (=1-FPR) is represented. This shouldn't be affected by the population distribution so much.