I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung diseases. To be clear (because this comes up a lot), each doctor evaluated the patients on their own without consultation. They were asked a variety of questions:
- first choice diagnosis
- Probability of IPF (a specific lung disease)
- Probability of HP (a specific lung disease)
- Probability of NSIP (a specific lung disease)
- Probability of CTD (a specific lung disease)
- Would you biopsy the patient?
- Would you treat or observe the patient?
- What treatment would you use?
Therefore, every doctor has answered these questions, on every patient.
Some of the doctors are from Universities and some from non-academic centres. Within each group, some of the doctors attend MDMs weekly (a training meeting) and some do not. We used the Kappa coefficient to calculate observer agreement between each unordered pair of doctors in each group. This is a standard approach to assessing agreement on decision making in my specialty
The results are as follows (all values are Medians and IQR of Kappas between every pair combination of doctors in the group)
We the looked to see if there were statistical differences in agreement between doctors who do and do not attend MDMs. We want to see if MDMs have a genuine training effect on the doctors, manifesting in increased agreement.
As you can see, based on p-values, there does seem to be a difference in agreement across all decisions (in Uni doctors) and 6/8 decisions (in non-academic doctors). From a practicality and clinical relevance perspective, the actual differences in Kappas are small (or tiny). But what we find most impressive is that the result does appear to be consistent across nearly all of the decisions made. We are aware that in a review of a manuscript reporting this data, a reviewer will likely hone in on the fact the differences are small and we wish to confront this issue head-on by not making wild claims based on individual decisions but to show statistically that MDMs improve agreement overall (in laymans terms, if it were a chance result one might expect to see improved agreement in a few of the decisions but not almost all).
We are unsure how this could be achieved. Some assistance would be great.