0
$\begingroup$

I have a question about inter-rater agreement using the ICC.

I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a personality profile -, on a variety of interval-scaled items (which, in my understanding are the "cases"). I am using an ICC(2,1), absolute agreement, as I would like to generalize to other raters from the same population (i.e., the general population).

In my analysis, I noticed that the ICC always decreased when less items are rated.

Lets say my 50 raters first rated the character on how important 10 social values are to them -- the ICC was quite good!

But when the 50 raters rated the assumed health of the character on 2 items (mental health and physical health), the ICC was close to zero.

In the latter case, both variables were highly correlated, which might have to do with it.

So, my questions are:

  1. Is it correct that a lower number of cases (or a too low number) is detrimental to the ICC.
  2. Is it correct that a high intercorrelation between items (cases) has negative effects on the ICC?
  3. If that is the case, would it make sense to create mean scores of highly correlated variables and then use an ICC(2,k)? (though, in my example, this would not work as it would leave me with a single item (case), which the ICC does not compute for.

Thanks for your help!

$\endgroup$
0
$\begingroup$

If you have 50 raters all rate one attribute for 100 characters, then you could set up a 100x50 character-by-rater matrix and use the standard ICC formulation to assess the inter-rater reliability for that attribute across characters. This could then be repeated for more attributes, one-by-one.

If you have 50 raters all rate 10 attributes for 1 character (which I think is what you're asking), then you could set up a 10x50 attribute-by-rater matrix and use the standard ICC formulation to assess the inter-rater reliability for that character across attributes. But note that, because ICC is based on variance component estimates, you will likely need more than 2 attributes for this to be trustworthy - so I don't think the issue here is your attributes being correlated but rather that you have so few of them. In such a case, especially with relatively few rating options (e.g., likert scale), you might consider using a weighted index of chance-adjusted agreement instead of the ICC.

If you have 50 raters all rate 10 attributes from 100 characters, then you would need to use some kind of generalizability study to account for the multiple sources of variability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.