2
$\begingroup$

I'm conducting a study in which I'm comparing 15 radiologists evaluations on a dichotomous variable to a gold standard, in a sample with 58 patients.

Obviously I've calculated overall accuracy, sensitivity and specificity for each rater, generating 15 2x2 tables.

BUT HOW DO I REPORT THESES INDICES WITH ALL RATERS COMBINED?

I've been told that just computing an average accuracy is incorrect. I can of course report the range, but I've got one rater who is a much poorer radiologist then the others, so this could be misleading.

I've Googled this question many times and there seems to be no accepted approach to this. For example these FDA guidelines https://www.fda.gov/RegulatoryInformation/Guidances/ucm071148.htm state that "We assume throughout that your study data do not include multiple samples from single patients. If you do have such data, we recommend that you consult with FDA statisticians on the appropriate calculation methods." What are the appropriate calculation methods?

Improve this question
$\endgroup$
3
  • $\begingroup$ I know, this is rather far fetched, but have you ever found an appropriate solution to this problem? I'm running into the exact same issue and I was wondering how to solve it. Even a pointer would help. $\endgroup$
    – Dom42
    Commented Nov 24, 2020 at 13:22
  • $\begingroup$ @Dom42 , I invented my own method, for sensitivity, by adding all raters true positives and divided by the sum of all raters true positives and false negatives. Thus, if rater 1 had 11 TP and 3 FN, and rater 1 had 15 TP and 1 FN, I calculated (11+15)/(11+3+15+1)=86.7%. $\endgroup$
    – Rasmus Green
    Commented Aug 18, 2021 at 6:21
  • $\begingroup$ @Dom42 This is not equal to the average, which would be (78.5+93.75)/2=86.1%. I have no justification for this, and I don't remember were I got the inspiration. But in my paper I did not have any citations for this method. I guess I just found it intuitive. $\endgroup$
    – Rasmus Green
    Commented Aug 18, 2021 at 6:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.