0
$\begingroup$

I'm trying to run an intraclass-correlation (inter-rater agreement) for personality data I have collected. However, I work with animals and as such the data has been collected over a period of a year or so, I have, in some cases different numbers of raters per subject, and different raters (or more specifically, not all raters have rated all subjects; some raters have done some subjects, other raters have done other subjects (and some have done all)). So, I have 7-9 ratings per subject, with some overlap between raters and subjects, but not always.

Is this possible, and if so which ICC should I use?

Any advice would be gratefully received.

$\endgroup$
  • $\begingroup$ got any answer on it? $\endgroup$ – Mpizos Dimitris Mar 1 '17 at 13:52
  • $\begingroup$ Nothing so far! $\endgroup$ – Bruce Rawlings Mar 4 '17 at 21:55
0
$\begingroup$

I had the same problem. I think I could solve that by using the formulas informed by Bliese (2000). You basically run an one-way ANOVA using the group as the independent variable. Then use the mean square between (MSB), mean square within (MSW) informed in the ANOVA output. In the calculation of the ICC(1), as you have a varying number of raters, use the average number of raters as k.

https://www.kellogg.northwestern.edu/rc/workshops/mlm/Bliese_2000.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.