3
$\begingroup$

Scenario: comparison of 2 different binary classifiers

Both classifiers report sensitivity and specificity and number actually positive (P), but classifier 1 is tested on a dataset with prevalence 20%, classifier 2 is tested on an enriched sample with prevalence 50%. I prefer PPV to specificity in my context, because I'm much more interested in the classification of the positive class, and can plot both classifiers' results on precision recall curve (PRC) space rather than ROC space, however classifier 2 will have a higher PPV (precision) just by virtue of having a higher prevalence.

Is it valid then to say, provided that sensitivity and specificity of both classifiers do not change, then for prevalence of 10%, I can calculate a prevalence-adjusted PPV (keeping the ratios of TP/FN and TN/FP the same)?

$\endgroup$

1 Answer 1

0
$\begingroup$

You are correct that you need to adjust your results. The PRC space baseline will change because of the different underlying sample prevalences. Sort of comparing the two classifiers on a common test dataset (which would be ideal) we should look into adjusting our reported metrics; Abu-Akel et al. (2018) Mind the prevalence rate: overestimating the clinical utility of psychiatric diagnostic classifiers give a nice and succinct treatment on the matter. Note that they refer at as the "Bayes’ adjusted positive and negative predicted values for the prevalence of the condition in the population" is what you refer as "10%" in your example.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.