# R-squared or ICC or Kappa or pairwise t-test comparisons to compare ratings

I would like to compare the scores (0-100 scale) of students who are rated by three different examiners. The correct score is assigned by fourth examiner.

I would like to quantify how different the scores assigned by 1st, 2nd and 3rd examiner to the fourth examiner.

I'm contemplating between R-squared, ICC, Kappa and pairwise t-test comparisons. Which measure is a better measure for quantifying the difference. Is comparing means (t-test) a good idea in this scenario ?

I would like to further perform power analysis. I didn't calculate sample size before, so is it safe to perform post-hoc power analysis ? What are the things, I need to consider ?

It would be great, if you can provide some intuition behind your recommendation for both parametric and non-parametric situations.

My goal is to actually choose one of those 3 examiners, I would like to see who was more precise and followed the instructions provided on how to score an exam.

What you are looking for is agreement, not correlation. This is different: If comparing two sets of ratings from two raters (examiners), denote one set by $$x$$ the other $$y$$. If these ratings or scores satisfy $$y=x+1,$$ say, then the correlation will be 1 but there is no agreement; in fact there is a constant disagreement of $$|y-x| = 1$$. Any measurement which does not take that into account cannot be an index of agreement! One correlation coefficient which could be used is the concordance correlation coefficient: see https://en.wikipedia.org/wiki/Concordance_correlation_coefficient This can be very close to some measures called ICC (there are many). t-test comparisons could be useful, but remember in the model $$y_i = \alpha + \beta x_i + \epsilon_i$$ you want to test both $$\alpha=0$$ and $$\beta=1$$. This somehow, while it might be useful, seems artificial since it treats $$x$$ and $$y$$ differently, while the problem of agreement is really symmetric.