Multiple Sclerosis: study design to beta test a software I am a beta-tester of a software that is intended to help the radiologist to interpret MRI reading of Multiple Sclerosis (MS).
MS is a disease that, over time, could lead to new lesions, expanding previous lesions, stability, or regression of existing lesions of the white matter of the brain.
Normal radiologist routine is about comparing new studies to old ones and spot those characteristics. It may take a lot of time and concentration, and plenty of new softwares come out to speed up this process or to make it more accurate (many lesions may be too small or may be missed).
How would you design a study to show that this software helps the radiology? My first impression is that it speeds up the process by using an innovative comparison technique and it results also in more accuracy.
My plan is to review aroud 100 MRI studies dividing them, by reading previous reports in: "new lesions", "increasing size of old lesions", "stability" and "regression". Then re-read again the scans with aid of the software and check for differences in accuracy and speed.
Then check if the differences I get are statistically significant or not.
Do you think this study design is right for my purpouse? Or would you suggest something different to make it more robust? Thanks
 A: This is not a permissible study design because of the learning effect. With 100 scans, you will easily recognize a few select scans you already performed and be subject to the biases you already had. That is to say, your classification of the scans (albeit unassisted by software) is subject to your personal biases. Lastly, this design is problematic because there is no gold standard, unless you have biopsy or autopsy results which I think I'm safe to say just ain't gonna happen.
A simple approach to reducing the bias of learning effect is to employ a Latin square design and confirmation with three additional, separate radiologists.

*

*Randomly divide the 100 scans into 2 groups. Reviewer A reviews the first group of 50 without software assistance, and the second group of 50 with software assistance. Reviewer B reviews the first group with software assistance and the second group without software assistance.


*Reviewer C, blinded to which scans are assisted by software or not, confirms all scans from Reviewer A. Reviewer D, also blinded, confirms all scans from Reviewer B. You may even subdivide the two groups of 50 into another two, overall 4 groups of 25, and split them among each reviewer so that the results from Reviewer A and B are evenly split or "blocked" among Reviewer C and D, and blocked on use of software (or not), in case there is reviewer effect among C and D.


*Aside from obvious analyses such as time-to-review-completion (expedience), confirmation rate is the closest measure of accuracy you can have. A scan is confirmed when the confirmatory scan confirms the initial radiologist assessment. You can model agreement using Cohen's Kappa, or a simple McNemar's test. The balanced design should preclude any need for multivariate modeling.
