A pathologist friend came to me for help with the following question for a research project. The goal is to compare the effectiveness of three different diagnostic techniques. The data set is as follows: there are 50 different specimens, each specimen was evaluated by 4 pathologists, and 3 different instruments (ie 600 total diagnoses). Each case has a possible diagnosis of positive or negative, and the true results are known, as they have been independently determined. The success rate depends on both the quality of the instrument and the skill of the pathologist, and we cannot assume that the four pathologists have the same proficiency. Finally, even though each person measured the same specimen 3 times, they can be treated as independent measurements.
What are the appropriate tests for comparing effectiveness among the instruments?
ADDED: Lots of good info in the answers, thanks to both. Any thoughts on how ROC and randomized block compare/contrast?
I'm not sure I've digested it enough to know which method is "better". Since the results need to be communicated to a certain audience, it probably depends on which is more widely used among that audience.