0
$\begingroup$

I am running an algorithm to detect malfunctioning parts of a system. We are simulating malfunctioning parts of the system with different percentages such as 10%, 25%,etc. What I want to show is how well we predict malfunctioning parts.

For example, there are total of 100 system parts and 10 of them are malfunctioning. Our algorithm predicts 12 malfunctioning parts but in reality 8 of these predicted parts are malfunctioning. Hence, the precision of our model is 8/12=0.66 and the recall is 8/10=0.8. But, when the percentage of malfunctioning part is increased from 10 to 25, the precision will inevitably increase because we are dealing with another data partition for which we will have more true positives compared to false positives. So, precision here is not a good measure. We can still use recall for measuring the completeness of correct results. What I want to ask is should I use a metric like π‘“π‘Žπ‘™π‘ π‘’ π‘π‘œπ‘ π‘–π‘‘π‘–π‘£π‘’π‘  / (π‘“π‘Žπ‘™π‘ π‘’ π‘π‘œπ‘ π‘–π‘‘π‘–π‘£π‘’π‘  + π‘‘π‘Ÿπ‘’π‘’ π‘›π‘’π‘”π‘Žπ‘‘π‘–π‘£π‘’π‘ ) to measure how many of irrelevant items are annotated as malfunctioning? Do we have a name for this metric?

$\endgroup$
1
  • $\begingroup$ Do your algorithms produce numeric scoring? If so you can compute ROC-AUC and it is considered a good metric. $\endgroup$
    – Kota Mori
    Sep 8, 2021 at 11:32

1 Answer 1

0
$\begingroup$

As you point out, the precision metric tends to depend on the population distribution; The precision tends to be higher if the "true" ratio in the population is higher.

A possible alternative metric is the ROC-AUC (https://en.wikipedia.org/wiki/Receiver_operating_characteristic). If the outcome produces numeric scoring and the final output label is determined by some threshold, then you can obtain the ROC curve and the area under the curve is used as a performance measurement.

Also, When I looked at https://en.wikipedia.org/wiki/Sensitivity_and_specificity, I thought you may be interested in the balanced accuracy metric among the listed measurements: $BA = (TPR + TNR)/2$. This is similar to ROC-AUC in that the trade-off between TPR and TNR (=1-FPR) is represented. This shouldn't be affected by the population distribution so much.

$\endgroup$

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.