Skip to main content
Bumped by Community user
edited tags
Link
kjetil b halvorsen
  • 82.8k
  • 32
  • 201
  • 663
edited tags; edited title
Link
user1213055
  • 201
  • 1
  • 2
  • 5

Comparison of percentages at individual level rather than groups R-squared or ICC or Kappa or pairwise t-test comparisons to compare ratings

Source Link
user1213055
  • 201
  • 1
  • 2
  • 5

Comparison of percentages at individual level rather than groups

I would like to compare the scores (0-100 scale) of students who are rated by three different examiners. The correct score is assigned by fourth examiner.

I would like to quantify how different the scores assigned by 1st, 2nd and 3rd examiner to the fourth examiner.

I'm contemplating between R-squared, ICC, Kappa and pairwise t-test comparisons. Which measure is a better measure for quantifying the difference. Is comparing means (t-test) a good idea in this scenario ?

I would like to further perform power analysis. I didn't calculate sample size before, so is it safe to perform post-hoc power analysis ? What are the things, I need to consider ?

It would be great, if you can provide some intuition behind your recommendation for both parametric and non-parametric situations.

My goal is to actually choose one of those 3 examiners, I would like to see who was more precise and followed the instructions provided on how to score an exam.