I am trying to compare the reliability of two different assessment methods (method 1 and method 2). Each method assesses task performance for a single cohort of students (N = 50).
For method 1, student performance is rated by 4 judges (who vary across students). The assessment method assigns students a numeric score.
For method 2, student performance is rated by 5 judges (judges are fixed and consistent across students). The assessment method assigns a numeric score based solely on narrative comments.
I now want to compare the reliability of the two assessment methods.
I have calculated ICCs, but I am not sure how to best compare whether they are equivalent. I have 95% CIs, and can assess degree of overlap. Is there something more "sophisticate"?