The kappa tag has no wiki summary.
3
votes
0answers
38 views
Quadratic weighted kappa versus linear weighted kappa
When should I use quadratic weighted kappa or linear weighted kappa?
I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
2
votes
1answer
11 views
Macro and Micro Inter-Coder/Rater Agreement
I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in
...
0
votes
0answers
19 views
How to interpret the “Units independence” assumption of Cohens Kappa?
I am using Cohens Kappa to assess the reliability of three judges that have manually classified 3000 cases into belonging to a class (1) or not (0)
I know that Cohens Kappa relies on the three ...
1
vote
2answers
97 views
Strange values of Cohen's kappa
I have two raters who agree on 93% on the cases (two options: yes or no). However, when calculating cohens kappa through crosstabs in spss I get really strange outcomes like -0.42 with a sig. of ...
1
vote
1answer
147 views
Quadratic weighted kappa strength of agreement
In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical ...
1
vote
1answer
239 views
How to calculate error matrices, K hat and var(K hat)?
I have a very simple data set consisting of three columns: Ground Truth Canopy Class, Method 1 Canopy Class and Method 2 Canopy Class. Each row in the columns represents the canopy class (i.e. 1 ...
0
votes
0answers
54 views
quadratic kappa and cross validation
I am trying to classify text based on their quality (4 classes). The training material contains texts that have been manually scored.
I am using ten-fold cross validation in order to find the best ...
3
votes
2answers
175 views
Inconsistency of findings between Kappa and ICC for IRR study
I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects:
...
0
votes
0answers
34 views
What can I learn from a survey where the participants are asked to compare treatment and control conditions?
I recall reading a story about a giant beverage company releasing a "new and improved" version of one of their beverages. They got positive feedback with surveys, but neglected to ask participants ...
0
votes
0answers
156 views
How should I interpret the Kappa Fleiss test?
I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
71 views
Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?
For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
2
votes
0answers
51 views
What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?
I found a question that is related here, but it doesn't really goes on what I want to know.
I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
0
votes
0answers
148 views
Software to compute p-values and Cohen's kappa for 10 by 10 contingency table?
I have a 10 by 10 contingency table with most of my cells = 0, and my total cell count = 70. Is there a statistical package or online calculator which I can use to obtain p-values and Cohen's kappa?
4
votes
1answer
125 views
Is the power for a Kappa test the same as underlying z-test?
The Kappa ($\kappa$) test is a Z-test kind of test. If I am not very wrong, to compute the $\kappa$ test, we can just estimate the appropriate variance $\hat {var}(\hat\kappa)$ for the kappa statistic ...
27
votes
1answer
1k views
Computing Cohen's Kappa variance (and standard errors)
The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen [1] to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time.
My ...
2
votes
2answers
490 views
Adjusting kappa inter-rater agreement for prevalence
I am trying to calculate kappa scores for present/absent decisions made by two raters and I have heard that they can be adjusted for prevalence of the object of measurement.
Can anyone advise on how ...
4
votes
1answer
94 views
Ordered category agreement
I have data on two processes, where the process assigns elements into ordered bins. I am interested in testing for agreement between the processes. What is the best way to do this (R code)? Here is ...
3
votes
1answer
188 views
Inter-rater reliability with many non-overlapping raters
I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk.
88 different raters provided judgments for ...
4
votes
2answers
978 views
Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?
I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions.
Should ...
12
votes
2answers
6k views
Inter-rater reliability for ordinal or interval data
Which inter-rater reliability methods are most appropriate for ordinal or interval data?
I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...
