Which ICC model to use for computer vs. human agreement

Question

I have an agreement study with the following basic construction:

~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to calculate an intraclass correlation coefficient for the purposes of parameter tuning in the algorithm (i.e. calculate bootstrapped estimates of ICC for various parameters sets in a Grid Search in the training cohort --> minimize ICC and use these parameters in the test cohort).

However, the human raters are different for each item. By this, I mean that each item is only rated by one human, and this human is not the same for each item (there are 5 total human raters that split the items up). The algorithm, on the other hand, rates all of the images.

So, in some sense, this seems like a situation to use ICC(1,1) aka the one-way random effects model given that each item is rated by a different "set" of raters (i.e. human 1 + computer, human 2 + computer, human 3 + computer, etc.). On the other hand, I could also justify in my mind the use of ICC(2,1) aka the two-way random effects model given that the human raters used are simply a "random selection" of all possible raters in the population.

In either case I will be using a "single rater" model type (as there is only one rating by a human and one rating by the computer for each item), and an "absolute agreement" model given the clinical implications of the study.

Thanks so much for your input!

Answer 1

Since your $k$ observations per object of measurement differ in a systematic way (i.e., human vs. algorithm), you should preserve their ordering (e.g., column 1 is human and column 2 is algorithm) and use a two-way model. See page 31 of the article cited below.

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.