I would like to compare the scores (0-100 scale) of students who are rated by three different examiners. The correct score is assigned by fourth examiner.
I would like to quantify how different the scores assigned by 1st, 2nd and 3rd examiner to the fourth examiner.
I'm contemplating between R-squared, ICC, Kappa and pairwise t-test comparisons. Which measure is a better measure for quantifying the difference. Is comparing means (t-test) a good idea in this scenario ?
I would like to further perform power analysis. I didn't calculate sample size before, so is it safe to perform post-hoc power analysis ? What are the things, I need to consider ?
It would be great, if you can provide some intuition behind your recommendation for both parametric and non-parametric situations.
My goal is to actually choose one of those 3 examiners, I would like to see who was more precise and followed the instructions provided on how to score an exam.