1
$\begingroup$

I am trying to understand how exactly to calculate the CP CI for the overall diagnostic accuracy of a single diagnostic test given a disease with low prevalence.

Here is an example:

  • Prevalence: 2.0%

  • True Positive: 431

  • False Positive: 30

  • False Negative: 29

  • True Negative: 116

And the results without confidence intervals are:

  • Sensitivity: 93.7%

  • Specificity: 79.5%

  • Accuracy: 79.7%

The accuracy (overall diagnostic accuracy) is defined as:

  • Accuracy = Sensitivity * Prevalence + Specificity * (1 - Prevalence)

Using the F-distribution, the CP CI interval is given as:

formula

But I am not sure what to substitute for:

  • x: # of successes
  • n: # of trials

particularly as the overall diagnostic accuracy incorporates prevalence. This prevalence is not in my 2x2 confusion matrix, so I doubt I can use uncorrected values from the 2x2 to determine the accuracy CIs.

How would a confidence interval for overall diagnostic accuracy taking into account prevalence be calculated using this 2x2 table and prevalence data?

Thank you!

EDIT: Comments mentioned using the normal approximation which would be: normal

But this would raise the following issue:

  • If p^ = accuracy = (79.7%) and n = # of total tests = 431 + 30 + 29 + 116, how can we assume that the normal distribution is appropriate, particularly for such a low prevalence and sample size?
$\endgroup$
1
  • $\begingroup$ Not sure about the statistical validity, but the calculator here appears to calculate the Clopper-Pearson confidence intervals for accuracy and accounts for prevalence: medcalc.org/calc/diagnostic_test.php $\endgroup$
    – pw-314
    Commented Feb 18, 2022 at 17:41

1 Answer 1

2
$\begingroup$

Accuracy is not a single binomial proportion, so the Clopper-Pearson interval does not apply. In contrast to Normal and Poisson, the sum of two binomial random variables with different means is not (in general) a binomial.

You can use a confidence interval based on the Normal approximation, preferably after some transformation like logit. Or you could bootstrap -- it won't be much better than an appropriately transformed Normal approximation, but at least you don't need to work out the transformation.

There isn't a straightforward alternative: in contrast to the problem of a confidence interval for a single binomial proportion, this problem has nuisance parameters: it's a two-parameter problem and you want a confidence interval for one parameter, so for better accuracy than the Normal approximation you have to do something with the other parameter.

It principle it would be possible to reparametrise so that you had the accuracy and a second parameter orthogonal to it, and then do some sort of profile likelihood, but it isn't easy.

Update: I've just worked out a profile likelihood for the special case of so-called balanced accuracy, the average of sensitivity and specificity, and it seems to work well on a first look. The idea is to rewrite the likelihood in terms of the average and the difference of sensitivity and specificity, then maximise over the difference for each value of the average to get a profile likelihood just depending on the average, then do a likelihood-based confidence interval.

Of course, it would need more study before one could actually recommend it

$\endgroup$
2
  • $\begingroup$ Can you fill in with some citations or reasoning for why a Normal approximation is appropriate in these cases? $\endgroup$ Commented Jul 6, 2020 at 13:35
  • $\begingroup$ Yes, I was under the impression that a normal approximation would not be appropriate given the small sample size and probability? $\endgroup$ Commented Jul 6, 2020 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.