1
$\begingroup$

I came up with this paper with 765 citations! On page 2, it expresses an equation relating these metrics:

Accuracy
sensitivity (a.k.a Recall).
specificity (a.k.a 1 - type1 error rate)
prevalence (relative frequency of class 1)

Accuracy = (sensitivity) (prevalence) + (specificity) (1 - prevalence)

Following the equation, it says,

"However, it worth mentioning, the equation of accuracy implies that even if both sensitivity and specificity are high, say 99%, it does not suggest that the accuracy of the test is equally high as well."

No way I understand it! So I thought these should be correct. Can you help me to evaluate my thoughts:

1- If sensitivity and specificity are above x%, Accuracy is above x% for sure!

2-If Accuracy is above x%, both sensitivity and specificity may not be above x%! But one of them is more than accuracy if the other one is less!

3-For two different samples and models on them, it might be that accuracy is better in one, but sensitivity and specificity are both worse in it (Similar to Simpson's paradox)

$\endgroup$
2
  • 1
    $\begingroup$ Wikipedia article on confusion matrix is good as a reference. $\endgroup$
    – Roger V.
    Jan 26, 2022 at 7:31
  • $\begingroup$ @RogerVadim Thanks my friend! I am always referring to this page...but would appreciate it if you look into my question which is about "relationship between accuracy and other metrics and confusion about it" $\endgroup$
    – Ehsan Sh
    Jan 26, 2022 at 22:05

1 Answer 1

1
$\begingroup$

The sentence in this paper that follows the one that is cited is as follows:

In addition to sensitivity and specificity, the accuracy is also determined by how common the disease in the selected population. A diagnosis for rare conditions in the population of interest may result in high sensitivity and specificity, but low accuracy. Accuracy needs to be interpreted cautiously.

The results of a (positive) test with a high sensitivity (e.g. 99%) and high specificity (e.g. 99%) should be interpreted cautiously if the prevalence is very low (e.g. 0.001) because the positive predictive value is low.

The authors give the correct definition of accuracy referring to the confusion matrix. But, in the sentence giving advice on interpretation of a test, the authors seem to confuse "accuracy" with "positive predictive value" and use the term "accuracy" in the "plain English" or "dictionary" sense:

"the quality or state of being correct....."

If the positive predictive value of a test is low, then the result of the single test--interpreted as having made a diagnosis--may not be correct.

CONCRETE EXAMPLE

For a scenario (provided as an example) in which 100,000 people are tested in order to try diagnose a disease and the sensitivity of the test is 99%, the specificity is 99% and the (true) prevalence of the disease is 0.001 (or 0.1%), there are 100 people who truly have the disease and 99,900 who truly do not have the disease.

Of the 100 people who truly have the disease, 99 will have positive test (true positives) and 1 will have a negative test (a false negative). Of the 99,900 people who truly do not have the disease, 999 will have a positive test (false positives) and 98,901 will have a negative test (true negatives). Of the 1,098 people with a positive test, 99 are true positives (the “diagnosis” based on this test is correct based only on this test) and 999 are false positives (the “diagnosis” based on this test is incorrect based only on this test).

All of this can be framed in terms of Bayes theorem. But the example makes the point.

RESOURCE

The following resource is an excellent calculator for the testing scenario for a broad range of sensitivities, specificities, and prevalences:

https://kennis-research.shinyapps.io/Bayes-App/

IN PRACTICE, HOW TESTING IS DONE FOR LOW PREVALENCE SCENARIOS

In the setting where a condition has low prevalence, like the example, a single test would not be used to make a “diagnosis.” A better, follow-up test, would be used in those who have an initial positive test (because there are very few false negatives—only 1 this example—and the problem in the setting of low prevalence is to distinguish the true positives from the false positives).

CONFUSION

This is not the only published paper that confuses the confusion matrix and interpretation of medical diagnostic tests. Note that year of publication is 2010.

ANSWER TO QUESTION

You are correct that the accuracy of a test with a sensitivity of 99% and specificity of 99% will have an accuracy of 99% when calculated using the formula in the confusion matrix.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.