I have N number of patients each who could have 1 of 5 diseases (A, B, C, D, E). There is clinical information that may improve the accuracy of doctor diagnoses of these N patients. All diagnoses will be confirmed.

I want to test whether that information improves diagnostic accuracy. To do this, I am going to calculate the AUC for each diagnosis for a doctor before he has this clinical information and after. I will calculate the percent agreements for each set of diagnoses. I want to be able to show that the AUC for each disease increases 15%. Based on previous studies, lets say I know the doctors' concordance statistic, or $c$-statistic, before given the clinical information is 70 units and the SD is 2 units. I want a power of 0.9 at a significance level of 0.05.

I need to calculate the minimum sample size needed for this. In STATA, I have found that my N must be 46, randomized 1:1 to receive diagnoses without the new information and with the new information. My problem is exactly how to interpret this. This is really for the binary diagnosis of A versus not A. Therefore, for 5 different possibilities I will need 115 (5 x 23) of each diagnosis in each group.


1 Answer 1


I think in the last paragraph you stumbled onto the confusion I had from the outset. The AUC is not really a useful measure unless there is a continuous diagnostic score, like a PSA or a Framingham Risk Score, to evaluate over a range of possible diagnostic thresholds.

Since this is a study where diagnosis is measured in terms of a "yes/no" outcome, you are actually better poised to conduct a more practical study. Risk scores, and their AUCs, have no clinical relevance. They must be translated into an action: diseased/healthy, treat/don't treat, insulin/diet, mastectomy/lumpectomy and chemotherapy. If the outcome is already on the "yes/no" scale you are done. We use different metrics to evaluate diagnostic performance.

The Cohen's Kappa is a measure of interrater reliability. When the comparison group is a diagnostic confirmation, what we also call a gold standard, then it is also a measure of recall. In fact, if the disease statuses, A, B, C, D, E, are ordinal categories, the Cohen's Kappa has multicategory analogues if, say, a diagnosis of C is closer to a disease status D than a diagnosis of A. This is, for example, useful for cancer staging studies where Stage III and Stage IV cancers usually receive aggressive treatment.

There are tools and methods on sample size calculations for these measures elsewhere on the internet:



  • $\begingroup$ Arghhhh. This is really good advice and I have for a long time used Kappas and weighted Kappas where appropriate and then at a statistical consultation in my Uni, the statistician claimed that Kappas were not really considered useful for the performance of a diagnostic test. And proceeded to suggest I go with the traditional measures of Sens/Spec/AUC. $\endgroup$
    – GhostRider
    Apr 12, 2018 at 14:28
  • $\begingroup$ There is however one wrinkle I didn't mention. In reality, the clinicians won't be giving a yes/no answer - they will give diagnostic likelihoods (%) for all diseases summing to 100%. So for a single patient, a doctor may say, A-60%, B-25%, D-15%. From this data, you can apply thresholds to create AUC curves. But I agree with what you have said about the Kappa. $\endgroup$
    – GhostRider
    Apr 12, 2018 at 14:31
  • $\begingroup$ @GhostRider RE: Kappa as a diagnostic accuracy: I can see the statistician's point of view there. Nonetheless, there's a lot of literature using Kappa against a gold standard for diagnostic accuracy. Just scrape PubMed for over 5,000 examples. Another approach is just to report sensitivity and specificity and give interval estimates for those values. A caveat though: when we do these studies, we think of the disease status as random and the diagnosis as fixed. If the physicians report likelihoods, this doesn't hold. You need a repeated measure or a cross over design with a mixed model. $\endgroup$
    – AdamO
    Apr 12, 2018 at 14:34
  • $\begingroup$ Agreed - especially when there is no accepted Gold standard. Kappa is often used as a surrogate for accuracy, like in this: ncbi.nlm.nih.gov/pubmed/27180021 and this: ncbi.nlm.nih.gov/pubmed/28860269 both by me. Unfortunately I am having great difficulty getting stats advice for a grant application. $\endgroup$
    – GhostRider
    Apr 12, 2018 at 14:49
  • $\begingroup$ @GhostRider Depending on the funder, it's nigh impossible to get a grant without dedicated methods support. This would be someone with at least an MS or MPH in biostatistics or epidemiology. PhDs are weighted better. Since the grant would support their salary, it's reasonable to expect them to front load the work on the application. $\endgroup$
    – AdamO
    Apr 12, 2018 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.