Which inter-rater reliability test is best for binomial, categorial and continuous variables?

Question

I have two observers who both looked at several scans we performed in patients. They both looked at all scans, they were blinded to clinical data and each others measurements, and they scored several things:

• whether the scan was abnormal (1 or 0)
• how bad it was (from 1 to 5)
• and they measured something using computer software (continuous variable, measurements usually between 2.50 and 7.50)

Now I read my old statistics book, which tells me for categorical variables and 2 observers, I can use Cohen's kappa. Is this correct?

For continuous variables, the book says I should use the intraclass correlation coefficient, but if the two observers are not systematically different (as assessed by a paired T-test, i.e. one observer does not consistently measure higher or lower as the other), this would be practically the same as using the Pearson's correlation coefficient. Is this also correct?

And what test should I use for the binomial variable?

Thanks for your help.

Answer 1

You can use a weighted chance-adjusted index of agreement ($\kappa$, $\pi$, or $S$) for ordered and unordered categories; you just need to use a different weighting scheme for each. For unordered (nominal) categories, you should use nominal or identity weights. For ordered (ordinal) categories, you should use linear or quadratic weights. For continuous measurements, you can use an intraclass correlation coefficient (ICC) or some form of error measurement. ICC is actually different from Pearson's r in that the latter is a linearity index while the former may be an additivity or agreement index depending on the formulation chosen. See this link for more information.