I have an agreement study with the following basic construction:
~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to calculate an intraclass correlation coefficient for the purposes of parameter tuning in the algorithm (i.e. calculate bootstrapped estimates of ICC for various parameters sets in a Grid Search in the training cohort --> minimize ICC and use these parameters in the test cohort).
However, the human raters are different for each item. By this, I mean that each item is only rated by one human, and this human is not the same for each item (there are 5 total human raters that split the items up). The algorithm, on the other hand, rates all of the images.
So, in some sense, this seems like a situation to use ICC(1,1) aka the one-way random effects model given that each item is rated by a different "set" of raters (i.e. human 1 + computer, human 2 + computer, human 3 + computer, etc.). On the other hand, I could also justify in my mind the use of ICC(2,1) aka the two-way random effects model given that the human raters used are simply a "random selection" of all possible raters in the population.
In either case I will be using a "single rater" model type (as there is only one rating by a human and one rating by the computer for each item), and an "absolute agreement" model given the clinical implications of the study.
Thanks so much for your input!