0
$\begingroup$

I am working on a research paper about reviewing people's expertise.

I have 21 respondents, each answered 10 questions. Now I have asked 4 groups of reviewers (each group consists of 3 people) to grade the answers between 1 and 5 (5 best). I have used Fless' Kappa to measure inter-rater reliability between reviewers in the same group.

Unfortunately one reviewer asked me to include a Bayesian hierarchical model to account for the variability among raters and to provide a probabilistic measure of rater consistency. This would allow for a more nuanced understanding of rater behavior and the inherent uncertainties in human judgment, which are oversimplified by the use of a linear scale in the current methodology.

In my opinion it is an over-complication. I wanted to check whether people from different backgrounds can assess people in the same topic. I do not see any benefits from using Bayesian hierarchical model for comparing reviewers.

I am trying to perform this test, but nonetheless I would like to respond to the reviewer POLITELY that this is not a use-case for this model and Fleiss kappa is sufficient.

Could you help me, or prove me wrong?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ If someone in the study wants to see a particular measurement, I see no good reason to refuse to supply it. As to its value in this particular case, it's hard to say much without knowing more details. Fleiss (not Fless) measures work best for simple binary observations. A diagnosis, say. It's a lot tougher to use them if you are, say, grading an English essay. For those, intrinsically subjective, ratings, probabilistic measures are indeed useful. But more information is usually better than less. $\endgroup$
    – lulu
    Commented Aug 7 at 20:36
  • $\begingroup$ Some related thoughts: stats.stackexchange.com/a/642532/69508 $\endgroup$
    – Galen
    Commented Aug 7 at 22:51
  • $\begingroup$ stats.stackexchange.com/a/641857/69508 $\endgroup$
    – Galen
    Commented Aug 7 at 22:51

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.