Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

learn more… | top users | synonyms

0
votes
0answers
4 views

Agreement Statistics with Multiple Panels

Consider a simple study where two raters are asked to classify 24 items as either "Yes" or "No", and where we're interested in the agreement between the two raters. This feels like a fairly ...
0
votes
0answers
9 views

Kappa for classifier evaluation with ground truth data

for a fraud detection (credit-scoring-like) problem I want to use kappa as an evaluation metric. Kappa has the nice property that it fits very well to a problem with highly imbalanced relation between ...
0
votes
1answer
21 views

Which measure of inter-rater agreement for multi-class rating

In our experiment subjects could choose up to three values from a list of 18 categorical values. So let's say that the classes are animals: cat, dog, mouse, sheep... And that the subjects were asked ...
1
vote
1answer
46 views

Classification/evaluation metrics for highly imbalanced data

I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations. http://blog.revolutionanalytics.com/2016/...
1
vote
1answer
38 views

Weighted Fleiss' Kappa for Interval Data

I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows: There are several (5-8) ...
0
votes
0answers
28 views

R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa

In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value ...
1
vote
1answer
73 views

Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and <...
0
votes
0answers
9 views

Inter annotator aggrement for unbalanced datasets - KAPPA statistics

Can you suggest a better Kappa to measure the agreement between two annotators for unbalanced datasets with two classes?
0
votes
1answer
22 views

Kappa2 zero value despite almost identical ratings

I have a dataset with self and informant ratings of a pathological outcome in 47 people. Despite the fact that the self and informant ratings are almost identical, kappa2 (irr) gives me a kappa value ...
0
votes
1answer
64 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
0
votes
0answers
22 views

Sample size calculation for weighted Kappa

I will highly appreciate if anyone let me know how to calculate sample size for weighted kappa. I do know how to do it for unweighted kappa.
1
vote
0answers
21 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
0
votes
0answers
26 views

Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
0
votes
1answer
34 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
0
votes
1answer
46 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
0
votes
0answers
18 views

small p and small kappa

I have a dataset of roughly 12,000 points. They are boolean values (1 or 0) for a "program" and "outcome". I am trying to find if the program has an impact on outcome. I did a logistic regression in ...
3
votes
1answer
94 views

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
1
vote
1answer
60 views

Does Cohen's Kappa sample size calculation need a check for “true kappa” feasibility?

The sample size computation require(irr) N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE) successfully runs and ...
0
votes
2answers
24 views

Cohen's Kappa, why not simple ratio

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ ...
0
votes
0answers
53 views

Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
1
vote
1answer
496 views

Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
2
votes
0answers
63 views

Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
1
vote
1answer
78 views

How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
0
votes
0answers
22 views

kappa, ICC, and combined score

I am running into a problem in trying to determine whether results from an xray matches results from an ultrasound. I have about 100 patients, and for each patient we measure a, b, c, and d using both ...
0
votes
1answer
61 views

How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
0
votes
0answers
48 views

Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
0
votes
0answers
22 views

How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
3
votes
0answers
77 views

Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
0
votes
0answers
237 views

Difference between ROC and Kappa and TSS

I am using species distribution model to model the absence-presence of species against a set of predictors. I was wondering if anyone can guide me with the following: My model gives me a high ROC ...
0
votes
1answer
16 views

Assessing the agreement between two tests

I have genetic data. I have two tests that pick up changes in the genome (amplifications and deletions) at several different points in the DNA. There are three basic outputs- high, normal, low. I want ...
0
votes
1answer
32 views

Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
0
votes
1answer
34 views

Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
1
vote
1answer
211 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
0
votes
0answers
26 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
0
votes
1answer
42 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
0
votes
1answer
181 views

How can this rating have negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: ...
2
votes
0answers
62 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
1
vote
1answer
53 views

How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
1
vote
0answers
118 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
0
votes
1answer
2k views

What is the intuition behind the Kappa statistical value in classification

I understand the formula behind the Kappa statistic value and how to calculate the O and E value from a confusion matrix. My question is what is the intuition behind this measure? Why does it work so ...
2
votes
1answer
223 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
1
vote
0answers
63 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
0
votes
1answer
68 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
3
votes
0answers
41 views

Is there such a thing as a multidimensional kappa test?

I am involved with designing a new diagnosis method for a routine clinical procedure. I want to compare the agreement between operators to the agreement between methods. Is there such a thing as a ...
2
votes
0answers
193 views

Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
0
votes
1answer
63 views

How to calculate this kind of kappa?

A colleague and I are trying to annotate an alignment task. We have two documents. The first document is the original copy and the other document is a modified version. The other annotator and I ...
1
vote
0answers
46 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
0
votes
0answers
473 views

Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where T'...
2
votes
1answer
130 views

Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and ...
48
votes
1answer
28k views

Kappa statistic in plain English?

I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked ...