I have a specific performance assessment metric which I aim to get some reliability and validity data for.
And I really would appreciate some help for the reliability:
In my research, running and assessing "a case" is extremely resource demanding for both me (the researcher) and the assessor. It is a tool that I have where I want to assess its inter-rater reliability.
However, there is only one case. E.g., a test which 8 coders (in each group) are grading.
I have defined the golden assessment where the criterion validity, or tool accuracy, will be assessed based on how close each group mean is to the golden assessment. (Content validity is covered theoretically).
However, I am struggling with providing data for the reliability of the tool. So.
I am looking for a statistic/method that allows me to calculate the coder's agreement where, as mentioned up top, I only have 1 case that all 8 assessors (each group) evaluate.
Coder's are split into two groups: gr1: coder's using the "control tool": there are three topics they are assessing where each scale item is from 0-100.
Here, I was thinking intra-class correlation random-two way. However, 1 case also makes this difficult.
gr2: Coder's that are using the "experiment tool". Here there are plenty of scale items; however, each item is binary and ordinal (yes/no).
In my mind, I was thinking Krippendorf for inter-rater agreement. However, only 1 case/test to evaluate makes this difficult.
Any suggestions for how to properly assess the reliability of the tool is appreciated as I see my suggested methods does not work.