Test for treatment change

Question

I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding to a given treatment. The new diagnostic tool is then used, and the patients are diagnosed and classified again. Patients are then classified into two categories:

1. the suggested treatment remains the same; or
2. the suggested treatment is changed.

It seems as if this should be a simple question, but still I can't figure out what kind of test I could use to see if the diagnostic tool provides any additional information leading to revision of treatment. Given that there is a test, I would also like to do some sample size calculations for the next part of the study.

Answer 1

You can think of the two different diagnostic tools as different 'raters', and assess the interrater agreement. The prototypical analysis is Cohen's kappa. Jon Uebersax has a webpage dedicated to interrater agreement with lots of good information; you'll probably want to spend some time there learning more about it.

If you use R, there is the irr package for assessing interrater agreement. It has functions for calculating Cohen's kappa (?kappa2), and for calculating power / sample size (?N2.cohens.kappa).