5
$\begingroup$

I came across a paper which reported the following results

Accuracy Specificity Sensitivity
97.49% 93.6% 94.3%

It seems unusual for accuracy to be higher than both sensitivity and specificity. Is this an error in reporting?

I looked at this question which suggests that this is possible for precision and recall, but I reproduced the test and found no examples in which accuracy exceeded both metrics.

I believe it should be possible to demonstrate this algebraically or even just logically (just to satisfy myself that the intuition is correct) but I haven't been able to do so.

For reference, this is the paper in question

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Does the paper report metrics for binary classification or multiclass? If the latter, how were the metrics calculated? $\endgroup$
    – Tim
    Commented Apr 2, 2023 at 21:41
  • $\begingroup$ @Tim it's a little unclear from the paper. The overall task is to predict "malignant" vs "benign" lesions. They show ROC curves for three different types of benign class vs the malignant class individually though it appears the classifier used is a binary one (and this is just a post-hoc analysis of performance on different types of data). As far as I can see there's no information on how the headline figures (second last page) have been calculated. I've edited the question to include the link to the paper. Would it make sense if this were a multi-class problem? $\endgroup$
    – Jack O'Neill
    Commented Apr 3, 2023 at 13:17
  • 1
    $\begingroup$ It does matter, as there are multiple ways how those metrics can be calculated for multi-class problems and some of those approaches are rather bad, see e.g. stats.stackexchange.com/questions/523695/… $\endgroup$
    – Tim
    Commented Apr 3, 2023 at 13:26
  • $\begingroup$ Thanks for that. It seems like I'd need more information to reliably interpret the results. If you'd like to add your comment as an answer I'll mark it accepted. $\endgroup$
    – Jack O'Neill
    Commented Apr 8, 2023 at 8:36

1 Answer 1

Reset to default
2
$\begingroup$

Here are "standard" definitions of sensitivity, specificity and accuracy from Wikipedia (as Tim writes, these are not set in stone):

$$ \text{Sensitivity}=\frac{TP}{P}, \quad \text{Specificity} = \frac{TN}{N}, \quad \text{Accuracy}=\frac{TP+TN}{P+N}, $$

where $P$ and $N$ are the numbers of positive and negative instances, and $TP$ and $TN$ are the numbers of true positive and true negative classifications.

Let's call the following the "sensitivity-specificity-accuracy inequality": if sensitivity is not equal to specificity, then accuracy is strictly between the two. (That is, your source either has an error, or uses some other definition of the three KPIs.)

Here is a proof without words:

sensitivity-specificity-accuracy inequality

And here are a few words to accompany the picture:

  • Sensitivity is the slope of the line connecting the origin to the point $(P, TP)$.
  • Specificity is the slope of the line connecting the origin to the point $(N, TN)$.
  • Accuracy is the slope of the line connecting the origin to the point $(P+N, TP+TN)$.

And since both $(P, TP)$ and $(N, TN)$ are in the upper right quadrant (possibly including the horizontal axis) and $(P+N, TP+TN)$ is the vector sum of the two vectors pointing to these coordinates, the "accuracy slope" must be strictly between the "sensitivity slope" and the "specificity slope".

Note that all three KPIs suffer from major problems: Why is accuracy not the best measure for assessing classification models?

R code for the picture (which I am not proud of in aesthetic terms):

p <- 3
tp <- 2
n <- 5
tn <- 4

opar <- par(mai=c(.5,.5,.1,.1))
    plot(c(0,p+n),c(0,tp+tn),type="n",las=1,xaxt="n",yaxt="n",xlab="",ylab="",bty="n")
    points(p,tp,pch=19)
    lines(c(0,p),c(0,tp))
    points(n,tn,pch=19)
    lines(c(0,n),c(0,tn))
    points(p+n,tp+tn,pch=19)
    lines(c(0,p+n),c(0,tp+tn),lty=2)
    arrows(0,0,p+n,0)
    arrows(0,0,0,tp+tn)
    axis(1,c(p,n,p+n),c("P","N","P+N"),tick=FALSE,line=-1)
    axis(2,c(tp,tn,tp+tn),c("TP","TN","TP+TN"),tick=FALSE,line=-1,las=1)
par(opar)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.