Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

learn more… | top users | synonyms

0
votes
0answers
4 views

R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa

In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value ...
1
vote
1answer
27 views

Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and ...
0
votes
1answer
16 views

Kappa2 zero value despite almost identical ratings

I have a dataset with self and informant ratings of a pathological outcome in 47 people. Despite the fact that the self and informant ratings are almost identical, kappa2 (irr) gives me a kappa value ...
0
votes
1answer
40 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
0
votes
1answer
37 views

How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
0
votes
1answer
12 views

Assessing the agreement between two tests

I have genetic data. I have two tests that pick up changes in the genome (amplifications and deletions) at several different points in the DNA. There are three basic outputs- high, normal, low. I want ...
0
votes
0answers
6 views

Inter annotator aggrement for unbalanced datasets - KAPPA statistics

Can you suggest a better Kappa to measure the agreement between two annotators for unbalanced datasets with two classes?
0
votes
1answer
26 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
0
votes
1answer
29 views

Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
0
votes
1answer
27 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
1
vote
1answer
33 views

How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
0
votes
2answers
21 views

Cohen's Kappa, why not simple ratio

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ ...
2
votes
1answer
345 views

How should I interpret Fleiss' kappa when it equals NaN?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
3
votes
0answers
206 views

How do I calculate reliability between an algorithm and multiple human experts?

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
1
vote
1answer
41 views

How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings ...
1
vote
1answer
245 views

Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
4
votes
1answer
528 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
0
votes
0answers
11 views

Sample size calculation for weighted Kappa

I will highly appreciate if anyone let me know how to calculate sample size for weighted kappa. I do know how to do it for unweighted kappa.
3
votes
1answer
53 views

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my ...
2
votes
1answer
712 views

Sample size needed for Fleiss' Kappa?

A group of raters (about 20) will be watching a series of videos and will be classifying them into 4 categories. I will be running a Fleiss' kappa to measure the agreement. How does one compute for ...
0
votes
1answer
27 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
1
vote
0answers
15 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
0
votes
1answer
25 views

Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
1
vote
1answer
156 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
0
votes
0answers
26 views

Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
0
votes
0answers
15 views

small p and small kappa

I have a dataset of roughly 12,000 points. They are boolean values (1 or 0) for a "program" and "outcome". I am trying to find if the program has an impact on outcome. I did a logistic regression in ...
1
vote
1answer
50 views

Does Cohen's Kappa sample size calculation need a check for “true kappa” feasibility?

The sample size computation require(irr) N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE) successfully runs and ...
37
votes
1answer
4k views

Computing Cohen's Kappa variance (and standard errors)

The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen [1] to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time. My ...
0
votes
0answers
41 views

Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
1
vote
1answer
174 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
2
votes
0answers
45 views

Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
43
votes
1answer
24k views

Kappa statistic in plain English?

I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked ...
0
votes
0answers
17 views

kappa, ICC, and combined score

I am running into a problem in trying to determine whether results from an xray matches results from an ultrasound. I have about 100 patients, and for each patient we measure a, b, c, and d using both ...
0
votes
0answers
31 views

Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
0
votes
1answer
121 views

How can this rating have negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: ...
0
votes
0answers
20 views

How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
3
votes
0answers
62 views

Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
0
votes
0answers
171 views

Difference between ROC and Kappa and TSS

I am using species distribution model to model the absence-presence of species against a set of predictors. I was wondering if anyone can guide me with the following: My model gives me a high ROC ...
1
vote
2answers
580 views

Quadratic weighted kappa strength of agreement

In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical ...
0
votes
0answers
18 views

Comparing agreement levels

I have a dataset with gold-standard measures and with an alternative method in two different anatomical sites (the data is ordered cathegorical). Intuitively from the data, the alternative method ...
0
votes
0answers
98 views

Kappa vs McNemar

I have this matrix: a b <-- classified as 4925 251 | a = YES 249 8 | b = NO And I'm trying to compute Kappa and McNemar statistics, but my results are very strange: Kappa ...
0
votes
0answers
22 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
2
votes
1answer
118 views

Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and ...
0
votes
0answers
51 views

Model Performance - Accuracy not significant but Kappa is?

I am comparing an SVM model to a logistic regression model. When I run a comparison between these models, I get the following output: Accuracy: SVM vs Logistic: p=0.7882 Kappa : SVM vs Logistic: ...
2
votes
0answers
51 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
2
votes
1answer
343 views

Limitations of Cohen's kappa for sparse data?

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, ...
6
votes
2answers
441 views

Inter-rater reliability with many non-overlapping raters

I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
1
vote
0answers
112 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
0
votes
1answer
1k views

What is the intuition behind the Kappa statistical value in classification

I understand the formula behind the Kappa statistic value and how to calculate the O and E value from a confusion matrix. My question is what is the intuition behind this measure? Why does it work so ...
1
vote
0answers
61 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...