I'm conducting a study in which I'm comparing 15 radiologists evaluations on a dichotomous variable to a gold standard, in a sample with 58 patients.
Obviously I've calculated overall accuracy, sensitivity and specificity for each rater, generating 15 2x2 tables.
BUT HOW DO I REPORT THESES INDICES WITH ALL RATERS COMBINED?
I've been told that just computing an average accuracy is incorrect. I can of course report the range, but I've got one rater who is a much poorer radiologist then the others, so this could be misleading.
I've Googled this question many times and there seems to be no accepted approach to this. For example these FDA guidelines https://www.fda.gov/RegulatoryInformation/Guidances/ucm071148.htm state that "We assume throughout that your study data do not include multiple samples from single patients. If you do have such data, we recommend that you consult with FDA statisticians on the appropriate calculation methods." What are the appropriate calculation methods?