Tagged Questions

Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

221 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
4 views

Agreement Statistics with Multiple Panels

Consider a simple study where two raters are asked to classify 24 items as either "Yes" or "No", and where we're interested in the agreement between the two raters. This feels like a fairly ...
7 views

Kappa for classifier evaluation with ground truth data

for a fraud detection (credit-scoring-like) problem I want to use kappa as an evaluation metric. Kappa has the nice property that it fits very well to a problem with highly imbalanced relation between ...
21 views

Which measure of inter-rater agreement for multi-class rating

In our experiment subjects could choose up to three values from a list of 18 categorical values. So let's say that the classes are animals: cat, dog, mouse, sheep... And that the subjects were asked ...
43 views

Classification/evaluation metrics for highly imbalanced data

I deal with a fraud detection (credit-scoring-like) problem. As such there is a highly imbalanced relation between fraudulent and non-fraudulent observations. http://blog.revolutionanalytics.com/2016/...
363 views

How should I interpret Fleiss' kappa when it equals NaN?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
814 views

Sample size needed for Fleiss' Kappa?

A group of raters (about 20) will be watching a series of videos and will be classifying them into 4 categories. I will be running a Fleiss' kappa to measure the agreement. How does one compute for ...
209 views

Cohen's Kappa using (irr) and kappa2() outputs NaN

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2(). For the ...
38 views

Weighted Fleiss' Kappa for Interval Data

I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows: There are several (5-8) ...
589 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
27 views

R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa

In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value ...
71 views

Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and <...
22 views

Kappa2 zero value despite almost identical ratings

I have a dataset with self and informant ratings of a pathological outcome in 47 people. Despite the fact that the self and informant ratings are almost identical, kappa2 (irr) gives me a kappa value ...
41 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
57 views

How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
16 views

Assessing the agreement between two tests

I have genetic data. I have two tests that pick up changes in the genome (amplifications and deletions) at several different points in the DNA. There are three basic outputs- high, normal, low. I want ...
9 views

Inter annotator aggrement for unbalanced datasets - KAPPA statistics

Can you suggest a better Kappa to measure the agreement between two annotators for unbalanced datasets with two classes?
63 views

How can I calculate percent agreement and kappa for each category?

My aim is to look at the reliability of a coding scheme which employs 7 exhaustive and mutually exclusive categories to categorise 40 "subjects" (i.e. a typical reliability study). More than 2 judges ...
33 views

Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
44 views

Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
52 views

How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
24 views

Cohen's Kappa, why not simple ratio

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ ...
209 views

How do I calculate reliability between an algorithm and multiple human experts?

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
75 views

How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
479 views

Power analysis for inter-rater reliability study (Kappa) with multiple raters

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of ...
22 views

Sample size calculation for weighted Kappa

I will highly appreciate if anyone let me know how to calculate sample size for weighted kappa. I do know how to do it for unweighted kappa.
93 views

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Stats newbie here, please bear with me. I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/...
33 views

Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
21 views

Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
32 views

Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
26 views

Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
18 views

small p and small kappa

I have a dataset of roughly 12,000 points. They are boolean values (1 or 0) for a "program" and "outcome". I am trying to find if the program has an impact on outcome. I did a logistic regression in ...
60 views

Does Cohen's Kappa sample size calculation need a check for “true kappa” feasibility?

The sample size computation require(irr) N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE) successfully runs and ...
4k views

Computing Cohen's Kappa variance (and standard errors)

The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen [1] to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time. My ...
51 views

Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
63 views

Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
28k views

Kappa statistic in plain English?

I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked ...
22 views

kappa, ICC, and combined score

I am running into a problem in trying to determine whether results from an xray matches results from an ultrasound. I have about 100 patients, and for each patient we measure a, b, c, and d using both ...
48 views

Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
177 views

How can this rating have negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: ...
22 views

How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
77 views

Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
233 views

Difference between ROC and Kappa and TSS

I am using species distribution model to model the absence-presence of species against a set of predictors. I was wondering if anyone can guide me with the following: My model gives me a high ROC ...
609 views

Quadratic weighted kappa strength of agreement

In the case of the kappa-value there are some attempts to qualify how good or bad the agreements are. For example Landis & Koch in the article The Measurement of Observer Agreement for Categorical ...
26 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
129 views

Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and ...
61 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
384 views

Limitations of Cohen's kappa for sparse data?

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, ...