2
$\begingroup$

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric variable ?

  • Suppose, for example, that I have two teachers assessing students on a test and I need to check the teachers (aka, raters) reliability. Both teachers will provide numerical scores/grades for the tests.

As far as I'm concerned, both Cohen's Kappa and Weighted kappa are for categorical and ordinal variables (respectively, right?). I've seen approaches to use Cronbach's Alpha or intraclass correlation - ICC or even Pearson's correlation , would any of these be adequate or any other test?

This is more of a theorical question, but tips on calculating this in R Studio would be much appreciated as well. Thanks in advance.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Your question reminded me of this paper, which had human raters rate facial expressions using a joystick, so continuous movement equated with facial intensity. The authors used several strategies, including Krippendorf's a, Fleiss' k, ICC, and simple coefficient matching, so at the very least there is precedence for ICC on continuous data. $\endgroup$
    – jrcalabrese
    Commented Nov 14, 2022 at 18:49

1 Answer 1

Reset to default
3
$\begingroup$

As per literature (Analyzing Rater Agreement p.115-122), ICC is indeed the solution for the case where we can't relate the data as categorical. This comes alongside Cronbach's $\alpha$ (which is used for test-retest cases) and of course Spearman's correlation $\rho$.

This question provides a similar answer.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank you, @Spatzle, I've just read the chapter of the material you've recommended. Things are clearer now. Authors mention that Cronbach's α is a special case of ICC, but this is not clear to me yet, I've seen elsewhere that they'd be the same thing, I'm a bit confused. Any thoughts? Concerning correlation, as far as I'm concerned, the 'consistent' approach would equal Pearson's rho, right? $\endgroup$
    – Larissa Cury
    Commented Nov 15, 2022 at 19:46
  • 1
    $\begingroup$ @LarissaCury There are different formulations of the ICC that answer different questions. One of these formulations, often called ICC(C,k), is equivalent to Cronbach's alpha. But the other formulations will differ. $\endgroup$
    – Jeffrey Girard
    Commented Jan 3, 2023 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.