I am a beta-tester of a software that is intended to help the radiologist to interpret MRI reading of Multiple Sclerosis (MS).

MS is a disease that, over time, could lead to new lesions, expanding previous lesions, stability, or regression of existing lesions of the white matter of the brain. Normal radiologist routine is about comparing new studies to old ones and spot those characteristics. It may take a lot of time and concentration, and plenty of new softwares come out to speed up this process or to make it more accurate (many lesions may be too small or may be missed).

How would you design a study to show that this software helps the radiology? My first impression is that it speeds up the process by using an innovative comparison technique and it results also in more accuracy.

My plan is to review aroud 100 MRI studies dividing them, by reading previous reports in: "new lesions", "increasing size of old lesions", "stability" and "regression". Then re-read again the scans with aid of the software and check for differences in accuracy and speed. Then check if the differences I get are statistically significant or not.

Do you think this study design is right for my purpouse? Or would you suggest something different to make it more robust? Thanks

  • $\begingroup$ Not sure what your definition of success is: (1) Software imitates human result? [quicker, cheaper diagnosis] Or (2) software finds something important that human didn't (and how you would know when that's really the case)? [possibly more accurate diagnosis] $\endgroup$
    – BruceET
    Mar 1, 2022 at 6:17
  • $\begingroup$ @BruceET Software helps human in achieving the same result (diagnosis of state of the disease) quicker, and more accurately (it helps identify more subtle lesions that it's hard to identify in a standard fashion). Software doesn't identify lesions itself but it presents the MRI one over the other, like a "mask". $\endgroup$
    – KingBOB
    Mar 2, 2022 at 7:06
  • $\begingroup$ But I'm trying to discover what counts you have that might be analyzed in a statistical way. $\endgroup$
    – BruceET
    Mar 2, 2022 at 8:33
  • $\begingroup$ @BruceET I can double-read exams and hopefully discover lesions that weren't identified previously. So I can demonstrate that using the software detects more lesions than manually. $\endgroup$
    – KingBOB
    Mar 2, 2022 at 21:24

1 Answer 1


This is not a permissible study design because of the learning effect. With 100 scans, you will easily recognize a few select scans you already performed and be subject to the biases you already had. That is to say, your classification of the scans (albeit unassisted by software) is subject to your personal biases. Lastly, this design is problematic because there is no gold standard, unless you have biopsy or autopsy results which I think I'm safe to say just ain't gonna happen.

A simple approach to reducing the bias of learning effect is to employ a Latin square design and confirmation with three additional, separate radiologists.

  1. Randomly divide the 100 scans into 2 groups. Reviewer A reviews the first group of 50 without software assistance, and the second group of 50 with software assistance. Reviewer B reviews the first group with software assistance and the second group without software assistance.

  2. Reviewer C, blinded to which scans are assisted by software or not, confirms all scans from Reviewer A. Reviewer D, also blinded, confirms all scans from Reviewer B. You may even subdivide the two groups of 50 into another two, overall 4 groups of 25, and split them among each reviewer so that the results from Reviewer A and B are evenly split or "blocked" among Reviewer C and D, and blocked on use of software (or not), in case there is reviewer effect among C and D.

  3. Aside from obvious analyses such as time-to-review-completion (expedience), confirmation rate is the closest measure of accuracy you can have. A scan is confirmed when the confirmatory scan confirms the initial radiologist assessment. You can model agreement using Cohen's Kappa, or a simple McNemar's test. The balanced design should preclude any need for multivariate modeling.

  • $\begingroup$ Thanks for your precius answer! I just wanted to add some details that may result in modifying your opinion. I'm blinded to those particular 100 scans, since I haven't reviewed any of them (my colleagues did). Also it can be considered to make radiologist C review a scan that has already been reported by radiologist B, so there's no learning bias. Anything comes to your mind having a radiologist A (me, blinded to all exams), and B and C blinded to their respective scans? $\endgroup$
    – KingBOB
    Feb 28, 2022 at 22:05
  • $\begingroup$ Also, regarding gold standard, all the diagnoses are considered definite since the referring neurologist diagnosed MS on the basis of current clinical and lab criteria. So we don't get confounders. $\endgroup$
    – KingBOB
    Feb 28, 2022 at 22:06
  • $\begingroup$ Note that I already have the final report for those scans, so in your example, radiologists C and D job can be just retrived by looking back on their final report after radiologists A and B completed their review. $\endgroup$
    – KingBOB
    Feb 28, 2022 at 22:13
  • $\begingroup$ @KingBOB if you wish to claim that a technique improves accuracy, you need objective evidence of accuracy, and not just better agreement with other imperfect methods. This was widely discussed when cancer CAD was introduced to improve clinical diagnoses: it only increased negative biopsies. But in MS, you can't assess the % of beta plaques without autopsy. The issue can be likened more to missing data than to "confounding". $\endgroup$
    – AdamO
    Feb 28, 2022 at 22:14
  • $\begingroup$ Last thing: how do you suggest to select these 100 scans? choosing randomly? just get 100 scans that have been labeled as "new lesion reported".? $\endgroup$
    – KingBOB
    Feb 28, 2022 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.