2
$\begingroup$

Because sensitivity and specificity are typically estimated as binomial proportions (e.g. k = TP, n = TP+FN), we can use any of the methods used to estimate the confidence interval for binomial distributions to quickly calculate the CIs without using bootstrapping.

The CI of accuracy can also be quickly calculated using the same method by picking the values from the confusion matrix of binary classifier (i.e. k = TP+NP, n = N). However, this is not possible for balanced accuracy, which gives equal weight to sensitivity and specificity and can therefore not directly rely on the numbers of the confusion matrix, which are biased by prevalence (like accuracy). The formula for balanced accuracy is

$$ BACC = \frac {Sensitivity + Specificity}{2} $$

Hence, my thought is to simply use this formula for the lower and upper bounds of the CI. That is,

$$ \text{lower bound of BACC CI} = \frac {\text{lower bound of Sensitivity CI} + \text{lower bound of Specificity CI}}{2} $$

$$ \text{higher bound of BACC CI} = \frac {\text{higher bound of Sensitivity CI} + \text{higher bound of Specificity CI}}{2} $$

It makes a lot of intuitive sense and the values seem to make sense. However, I wondered if this is actually a sensible and sound method to calculate the CI of balanced accuracy.

$\endgroup$
0

3 Answers 3

4
$\begingroup$

I have been looking into this a bit more, and it seems as though a Normal confidence interval plus a logit transformation does very well in modest sample sizes.

As earlier, define $$\widehat{\mathrm{sens}}\sim N(\mathrm{sens}, \sigma^2)$$ and $$\widehat{\mathrm{spec}}\sim N(\mathrm{spec}, \tau^2)$$ then for balanced accuracy $$\widehat{\mathrm{bla}}\sim N\left(\mathrm{bla}, \frac{\sigma^2+\tau^2}{4}\right)$$

Now take a logit transformation

$$\mathrm{logit}(\widehat{\mathrm{bla}})\sim N\left(\mathrm{bla}, \frac{\sigma^2+\tau^2}{4\mathrm{bla}^2(1-\mathrm{bla})^2}\right)$$ compute a confidence interval $(l,\,u)$ for $\mathrm{logit}({\mathrm{bla}})$ using this Normal approximation, then transform back to the probability scale as $(\mathrm{expit}(l),\,\mathrm{expit}(u))$

$\endgroup$
2
  • $\begingroup$ Thanks a lot for your effort! To clarify: I assume the conf. int. should then be $ \mathrm{expit} \left( \mathrm{logit}(\mathrm{bla}) \pm 1.96 * \frac{\mathrm{logit}(s)}{\sqrt n} \right) $ and not $ \mathrm{bla} \pm \mathrm{expit} \left( 1.96 * \frac{\mathrm{logit}(s)}{\sqrt n} \right) $, whereby $ \mathrm{logit}(s) = \sqrt{\frac{\sigma^2+\tau^2}{4\mathrm{bla}^2(1-\mathrm{bla})^2}} $. Is that correct? Because in the latter case, the values would be clearly wrong. $\endgroup$
    – incurious
    Commented Jul 10, 2020 at 23:18
  • $\begingroup$ Yes, logit, then confidence interval, then expit $\endgroup$ Commented Jul 11, 2020 at 7:57
5
$\begingroup$

While I'm not at all convinced balanced accuracy is a useful summary, that's also not how you compute a confidence interval for it.

To a reasonable approximation, the estimated sensitivity and specificity will be Normally distributed around the true values.

If $$\widehat{\mathrm{sens}}\sim N(\mathrm{sens}, \sigma^2)$$ and $$\widehat{\mathrm{spec}}\sim N(\mathrm{spec}, \tau^2)$$ then for balanced accuracy $$\widehat{\mathrm{bla}}\sim N\left(\mathrm{bla}, \frac{\sigma^2+\tau^2}{4}\right)$$

You can compute $\sigma$ and $\tau$ by dividing the confidence interval lengths for sensitivity and specificity by $2\times 1.96$

$\endgroup$
8
  • $\begingroup$ Thank you! My assumption was that the normal distribution would be inappropriate here, am I wrong? $\endgroup$
    – incurious
    Commented Jul 1, 2020 at 21:33
  • $\begingroup$ You won't find anything better -- except that you might try the Normal distribution on a transformed scale such as logit(sensitivity) $\endgroup$ Commented Jul 1, 2020 at 21:38
  • $\begingroup$ So, despite being an estimate based on two binomially distributed estimate, this can't serve as the basis for calculating the CI, right? Also, am I right to assume that you always mean the standard distribution (whether you write $\sigma^2$ or $\sigma$)? And do you mind explaining why we are dividing by 4? Thanks! $\endgroup$
    – incurious
    Commented Jul 1, 2020 at 22:15
  • $\begingroup$ (+1) for CI of difference: 2nd parameter of normal distribution denoted here by $N$ is population variance. Also, throughout: $\sigma^2, \tau^2$ correctly denote variances of estimates, while $\sigma, \tau$ denote corresponding standard deviations. // Finally, $Var((X-Y)/2) = [Var(X) + Var(Y)]/4.$ $\endgroup$
    – BruceET
    Commented Jul 1, 2020 at 22:29
  • $\begingroup$ thank you @BruceET. I was slightly confused by the use of var. as I got used to s.d. being used for normal distributions. Just to make sure: the CI would be $(\bar{BACC} + 1.96 * \frac{\sqrt{\frac{\sigma^2 + \tau^2}{4}}}{\sqrt{10}}, \bar{BACC} - 1.96 * \frac{\sqrt{\frac{\sigma^2 + \tau^2}{4}}}{\sqrt{10}})$ (for 10 resamples), correct? $\endgroup$
    – incurious
    Commented Jul 1, 2020 at 22:59
1
$\begingroup$

Sensitivity and specificity are two entirely different measures of the usefulness of a test. One is based on a (presumably small) population of subjects who have the disease or condition; the other is based on a (presumably much larger) population of subjects who don't.

I can see no valid rationale for averaging the two. As an example, suppose a test has sensitivity 99% but its specificity is 1%, essentially rendering the test useless.

  • A bogus test that just declares 99% of subjects to be 'positive', absent all contact with reality, would do as well.

  • Then how could you justify a definition of 'test accuracy' to say the test is "50% accurate"?

Example: Consider a population of 100,000 with 5% prevalence so that 5000 have the disease and 95,000 do not. Especially in developmental stages it is not unrealistic for a test to have 95% sensitivity and 80% specificity.

Here are the consequences of testing everyone in the population:

  • 4900 correctly treated or quarantined due to true positive results, and 100 undetected potential 'spreaders' of the disease.
  • 19,000 incorrectly quarantined or treated (by whatever means) due to false positive results, and 76,000 with no direct consequences of testing.

Especially considering that any member of the population can get the disease at any point in time, the situation is sufficiently difficult that unjustified simplifications are not likely to be helpful.

$\endgroup$
2
  • $\begingroup$ Fair point! Of course, both sensitivity and specificity must be reported as well. And the measure is probably not very valid for highly imbalanced sensitivity and specificity values (but still an improvement over accuracy). However, in my case, true negatives are as important as true positives. Hence, such a measure as F1 that focuses on the positive side seemed inappropriate. $\endgroup$
    – incurious
    Commented Jul 1, 2020 at 21:40
  • $\begingroup$ Inevitably, this site must accommodate some measure of respectful disagreement. And I respectfully, completely disagree with the usefulness of such attempted 'simplifications' as BACC. $\endgroup$
    – BruceET
    Commented Jul 1, 2020 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.