R-squared or ICC or Kappa or pairwise t-test comparisons to compare ratings

Question

I would like to compare the scores (0-100 scale) of students who are rated by three different examiners. The correct score is assigned by fourth examiner.

I would like to quantify how different the scores assigned by 1st, 2nd and 3rd examiner to the fourth examiner.

I'm contemplating between R-squared, ICC, Kappa and pairwise t-test comparisons. Which measure is a better measure for quantifying the difference. Is comparing means (t-test) a good idea in this scenario ?

I would like to further perform power analysis. I didn't calculate sample size before, so is it safe to perform post-hoc power analysis ? What are the things, I need to consider ?

It would be great, if you can provide some intuition behind your recommendation for both parametric and non-parametric situations.

My goal is to actually choose one of those 3 examiners, I would like to see who was more precise and followed the instructions provided on how to score an exam.

Answer 1

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $x$ the other $y$. If these ratings or scores satisfy $$ y=x+1, $$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $|y-x| = 1$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$ y_i = \alpha + \beta x_i + \epsilon_i $$ you want to test both $\alpha=0$ and $\beta=1$. This somehow, while it might be useful, seems artificial since it treats $x$ and $y$ differently, while the problem of agreement is really symmetric.

Coefficients like kappa or Krippendorff's alpha are directly constructed for evaluating agreement, so are probably the most natural choice. Krippendorff's alpha is the more general of the two, and, indeed, it can be used with more than two raters. There is an R package irr which implements these and others. But these coefficients are numbers and do not indicate the nature of the disagreement, so plots (specifically the Tukey mean-difference plot is very useful, also called Bland-Altman plot How does one interpret a Bland-Altman plot?, or Agreement between methods with multiple observations per individual). Regression models as above can also be useful in understanding the nature of the disagreement.

A book-length treatment is Kilem L. Gwet: "Handbook of interrater reliability" which I just got from the library for a project, so in some time maybe I will come back with some more information.