# Tagged Questions

Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

4 views

### Agreement Statistics with Multiple Panels

Consider a simple study where two raters are asked to classify 24 items as either "Yes" or "No", and where we're interested in the agreement between the two raters. This feels like a fairly ...
9 views

### Kappa for classifier evaluation with ground truth data

for a fraud detection (credit-scoring-like) problem I want to use kappa as an evaluation metric. Kappa has the nice property that it fits very well to a problem with highly imbalanced relation between ...
21 views

### Which measure of inter-rater agreement for multi-class rating

In our experiment subjects could choose up to three values from a list of 18 categorical values. So let's say that the classes are animals: cat, dog, mouse, sheep... And that the subjects were asked ...
46 views

### Classification/evaluation metrics for highly imbalanced data

I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations. http://blog.revolutionanalytics.com/2016/...
38 views

### Weighted Fleiss' Kappa for Interval Data

I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows: There are several (5-8) ...
28 views

### R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa

In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value ...
73 views

### Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and <...
9 views

### Inter annotator aggrement for unbalanced datasets - KAPPA statistics

Can you suggest a better Kappa to measure the agreement between two annotators for unbalanced datasets with two classes?
22 views

### Kappa2 zero value despite almost identical ratings

I have a dataset with self and informant ratings of a pathological outcome in 47 people. Despite the fact that the self and informant ratings are almost identical, kappa2 (irr) gives me a kappa value ...
64 views

### How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
22 views

### Sample size calculation for weighted Kappa

I will highly appreciate if anyone let me know how to calculate sample size for weighted kappa. I do know how to do it for unweighted kappa.
21 views

### Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
26 views

### Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
34 views

### Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
46 views

### Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
18 views

### small p and small kappa

I have a dataset of roughly 12,000 points. They are boolean values (1 or 0) for a "program" and "outcome". I am trying to find if the program has an impact on outcome. I did a logistic regression in ...
94 views

### Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
60 views

### Does Cohen's Kappa sample size calculation need a check for “true kappa” feasibility?

The sample size computation require(irr) N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE) successfully runs and ...
24 views

### Cohen's Kappa, why not simple ratio

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ ...
53 views

### Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
496 views

### Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
63 views

### Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
78 views

### How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
22 views

### kappa, ICC, and combined score

I am running into a problem in trying to determine whether results from an xray matches results from an ultrasound. I have about 100 patients, and for each patient we measure a, b, c, and d using both ...
61 views

### How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
48 views

### Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
22 views

### How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences.Â Let's say we have sequence 1 and sequence 2 asÂ an example of the task: Sequence 1: Â A B C D E Â F ...
77 views

### Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
237 views

### Difference between ROC and Kappa and TSS

I am using species distribution model to model the absence-presence of species against a set of predictors. I was wondering if anyone can guide me with the following: My model gives me a high ROC ...
16 views

### Assessing the agreement between two tests

I have genetic data. I have two tests that pick up changes in the genome (amplifications and deletions) at several different points in the DNA. There are three basic outputs- high, normal, low. I want ...
32 views

### Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
34 views

### Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
211 views

### Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
26 views

### Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
42 views

### Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
181 views

### How can this rating have negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: ...
62 views

### Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
53 views

### How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
118 views

### When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
2k views

### What is the intuition behind the Kappa statistical value in classification

I understand the formula behind the Kappa statistic value and how to calculate the O and E value from a confusion matrix. My question is what is the intuition behind this measure? Why does it work so ...
223 views

### What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
63 views

### Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
68 views

### What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
41 views

### Is there such a thing as a multidimensional kappa test?

I am involved with designing a new diagnosis method for a routine clinical procedure. I want to compare the agreement between operators to the agreement between methods. Is there such a thing as a ...
193 views

### Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
63 views

### How to calculate this kind of kappa?

A colleague and I are trying to annotate an alignment task. We have two documents. The first document is the original copy and the other document is a modified version. The other annotator and I ...
46 views

### Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
473 views

### Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where T'...