6
$\begingroup$

A pathologist friend came to me for help with the following question for a research project. The goal is to compare the effectiveness of three different diagnostic techniques. The data set is as follows: there are 50 different specimens, each specimen was evaluated by 4 pathologists, and 3 different instruments (ie 600 total diagnoses). Each case has a possible diagnosis of positive or negative, and the true results are known, as they have been independently determined. The success rate depends on both the quality of the instrument and the skill of the pathologist, and we cannot assume that the four pathologists have the same proficiency. Finally, even though each person measured the same specimen 3 times, they can be treated as independent measurements.

What are the appropriate tests for comparing effectiveness among the instruments?

Thanks.

ADDED: Lots of good info in the answers, thanks to both. Any thoughts on how ROC and randomized block compare/contrast?

I'm not sure I've digested it enough to know which method is "better". Since the results need to be communicated to a certain audience, it probably depends on which is more widely used among that audience.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Good first question! I'm not convinced each person's three measurements on the same specimen can really be treated as independent though. Did they know (or were they likely to realise) that the specimen was the same? Did they know the 'true' results when they made the measurements? Also do you know how many of the 'true' results were positive and how many were negative? $\endgroup$
    – onestop
    Commented Mar 29, 2011 at 12:06
  • $\begingroup$ You're right to worry about the independence of the measurements, but I think they did a pretty good job designing the experiment to minimize any memory between measurements (they spread out the diagnoses over several sessions and would only run a randomized subset of specimens on a single instrument during a session). They did not know the true results at the time. I can find out what the positive result rate was, but I think it's roughly 50%. $\endgroup$
    – Gregg L
    Commented Mar 29, 2011 at 23:00
  • $\begingroup$ I don't know anything about ROC, but what little I've read (cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf) suggests that getting a significance measure is messy. $\endgroup$
    – Avery Richardson
    Commented Apr 24, 2011 at 17:13

2 Answers 2

Reset to default
5
$\begingroup$

As it is described in the original post, the experiment is a randomized block.

  • Pathologist (4 levels) is a blocking factor; the experiment is repeated within each pathologist.
  • Instrument (3 levels) and the true result (2 levels) of the test are the two treatments, which I assume were assigned randomly.
  • Consider the different specimens to be replications of each treatment combination.
  • The one response variable is whether the pathologist's diagnosis is correct (2 levels).

Because the result is categorical, the link function will need to be something like logit or probit. Here's some R code that does that. It may need to be extended depending on your friend's hypotheses.

library(lme4)
glmer(correctness ~ instrument*trueresult + (1|pathologist),
  family = binomial)

The coefficients from a logit model can be interpreted in relation to odds ratios. For a particular combination of predictors, the model estimates an odds ratio. The individual coefficients indicate how the odds ratio changes depending on the predictors.

If your friend doesn't care about distinguishing between type I and type II error, he or she can drop true result predictor from the model.

library(lme4)
glmer(correctness ~ instrument+trueresult + (1|pathologist),
  family = binomial)

The measurement in multiple sessions may be an incomplete block design, so your friend should look at those if he or she is concerned about the assumption of independence among measurements.

Improve this answer
$\endgroup$
2
$\begingroup$

The ROC curve (Receiver Operating Characteristics) is one of the techniques available. You can check the questions with the tag roc on this site for further details. The wikipedia article http://en.wikipedia.org/wiki/Receiver_operating_characteristic and the external links to it may also be useful. Some other methods can be found here http://onbiostatistics.blogspot.com/2011/01/agreement-statistics-and-kappa.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.