I am calculating the Intraclass Correlation coefficient (ICC) as per the method of Shrout-Fleiss (1979), using the (3, 1) model in which the judges are fixed as are the targets. Some of my ICC values are negative. Is this to be expected? How to interpret this? I understand that the maximum ICC is 1.0, but is there a minimum value? How are negative values interpreted?
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ "ICC can become negative when the within-groups variance exceeds the between-groups variance" - tx.liberal.ntu.edu.tw/~purplewoo/Literature/!DataAnalysis/… $\endgroup$– ttnphnsCommented Jun 6, 2014 at 21:03
-
1$\begingroup$ see also stats.stackexchange.com/q/180081/3277 $\endgroup$– ttnphnsCommented Nov 4, 2015 at 9:49
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You might want to read the original paper to get a sense of what your ICC statistic is doing, how it is constructed and what it means.
Apparently, the ICC can go negative, since the numerator involves a difference between two quantities. It probably means that you should use a different measure. I would estimate the between judge and within judge variation with a mixed effects model and look to see if there is a meaningful difference between the judges.
In my experience, when the math gives you something stupid (like a negative estimate for something that should be positive), it is because one is trying to estimate something that does not exist, or that makes no sense, or that the data do not support.
-
1$\begingroup$ Thanks. Interestingly I'm trying to use ICC to assess different nurse's ability to replicate a skin reaction on a patient. So the nice thing is that I have a physical measurement which is objective and easily measured (unlike many situation where ICC is used for raters of personality constructs, etc.). Therefore the reactions certainly exist; it just seems that in my case the variation within a subject is equally as large, if not larger, than the variation between the nurses. $\endgroup$– BosleyCommented Jun 9, 2014 at 17:15
-
1$\begingroup$ That's good -- I think. It suggests that the nurses are fairly reliable - although if the "within-nurse" variation is larger than you want, that could be a problem. $\endgroup$– PlacidiaCommented Jun 9, 2014 at 17:43