I have a question about inter-rater agreement using the ICC.
I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a personality profile -, on a variety of interval-scaled items (which, in my understanding are the "cases"). I am using an ICC(2,1), absolute agreement, as I would like to generalize to other raters from the same population (i.e., the general population).
In my analysis, I noticed that the ICC always decreased when less items are rated.
Lets say my 50 raters first rated the character on how important 10 social values are to them -- the ICC was quite good!
But when the 50 raters rated the assumed health of the character on 2 items (mental health and physical health), the ICC was close to zero.
In the latter case, both variables were highly correlated, which might have to do with it.
So, my questions are:
- Is it correct that a lower number of cases (or a too low number) is detrimental to the ICC.
- Is it correct that a high intercorrelation between items (cases) has negative effects on the ICC?
- If that is the case, would it make sense to create mean scores of highly correlated variables and then use an ICC(2,k)? (though, in my example, this would not work as it would leave me with a single item (case), which the ICC does not compute for.
Thanks for your help!