0
votes
0answers
7 views

Comparison test of reapeated Cohen's kappa

Suppose two raters judged a set of stimuli. Then Cohen's kappa was calculated as measure of inter-rater agreement ($\kappa_{pre-training}$). Suppose now that after a training session, the same two ...
0
votes
1answer
13 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
1
vote
0answers
23 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
1
vote
0answers
115 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
1
vote
0answers
141 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
2
votes
1answer
975 views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
1
vote
0answers
62 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
3
votes
0answers
266 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
2
votes
1answer
33 views

Macro and Micro Inter-Coder/Rater Agreement

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in ...
3
votes
2answers
474 views

Inconsistency of findings between Kappa and ICC for IRR study

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
1
vote
0answers
268 views

How should I interpret the Kappa Fleiss test?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
103 views

Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
2
votes
0answers
74 views

What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
2
votes
2answers
1k views

Adjusting kappa inter-rater agreement for prevalence

I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement. Can anyone advise on how ...
4
votes
1answer
120 views

Ordered category agreement

I have data on two processes, where the process assigns elements into ordered bins. I am interested in testing for agreement between the processes. What is the best way to do this (R code)? Here is ...
3
votes
1answer
289 views

Inter-rater reliability with many non-overlapping raters

I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
4
votes
2answers
2k views

Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?

I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions. Should ...
15
votes
2answers
10k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...