5
$\begingroup$

I'm new to inter rater reliability calculations.

I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example:

For example,

         dev1  dev2  dev3  dev4   dev5
ranker1   1     2     3
ranker2   3     1     2
ranker3         3     1     2
ranker4               1     3      2
ranker5   2     1           2
ranker6         3     1            2

The ratings measure the "level of expertise": "1" the best expertise, "2" less expertise, and "3" lowest expertise

My question is what the best algorithm to calculate the inter rater reliability among these raters?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Inter-rater reliability estimates become difficulty when you don't have complete overlap. In your example, all raters do not rate all developers. There is no rater that is common to all developers, and no developer that is common to all raters. The many-facet Rasch model was really designed for this type of analysis (incomplete overlap), and should be able to give you the information you require (estimates of rater variance, rater stringency/leniency, fit statistics, inter-rater agreement and estimates of rater reliability). If you data-set is not large, there are programs available for free (Big Steps or mRm in R for example).

Improve this answer
$\endgroup$
2
  • $\begingroup$ sounds good. Would you recommend some resources to read about it? $\endgroup$
    – Programmer
    Apr 23, 2013 at 18:15
  • $\begingroup$ Sure. rasch.org/rmt/contents.htm winsteps.com/minifac.htm Setting up the specification files can be awkward. Let me know if you want me to run it for you and send you the output. The data would need to be in a csv file or an Excel file. Good luck with this - it can be very interesting. $\endgroup$
    – doug.numbers
    Apr 23, 2013 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.