I have an agreement study with the following basic construction:

~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to calculate an intraclass correlation coefficient for the purposes of parameter tuning in the algorithm (i.e. calculate bootstrapped estimates of ICC for various parameters sets in a Grid Search in the training cohort --> minimize ICC and use these parameters in the test cohort).

However, the human raters are different for each item. By this, I mean that each item is only rated by one human, and this human is not the same for each item (there are 5 total human raters that split the items up). The algorithm, on the other hand, rates all of the images.

So, in some sense, this seems like a situation to use ICC(1,1) aka the one-way random effects model given that each item is rated by a different "set" of raters (i.e. human 1 + computer, human 2 + computer, human 3 + computer, etc.). On the other hand, I could also justify in my mind the use of ICC(2,1) aka the two-way random effects model given that the human raters used are simply a "random selection" of all possible raters in the population.

In either case I will be using a "single rater" model type (as there is only one rating by a human and one rating by the computer for each item), and an "absolute agreement" model given the clinical implications of the study.

Thanks so much for your input!


Since your $k$ observations per object of measurement differ in a systematic way (i.e., human vs. algorithm), you should preserve their ordering (e.g., column 1 is human and column 2 is algorithm) and use a two-way model. See page 31 of the article cited below.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.