# Tagged Questions

Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

776 views

### Sample size needed for Fleiss' Kappa?

A group of raters (about 20) will be watching a series of videos and will be classifying them into 4 categories. I will be running a Fleiss' kappa to measure the agreement. How does one compute for ...
354 views

### How should I interpret Fleiss' kappa when it equals NaN?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
30 views

### Weighted Fleiss' Kappa for Interval Data

I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows: There are several (5-8) ...
54 views

### Get a 95% confidence interval for Cohen's Kappa in R

I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and <...
56 views

### Does Cohen's Kappa sample size calculation need a check for “true kappa” feasibility?

The sample size computation require(irr) N.cohen.kappa(rate1 = 0.9, rate2 = 0.5, k1 = 0.7, k0 = 0.5, alpha=0.05, power=0.8, twosided=FALSE) successfully runs and ...
58 views

### How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (...
28 views

### Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders

I have some trouble calculating the inter-rater reliability for a nominal (0/1) variable. The study design was as follows: different subjects were rated by different subsets of coders, in which each ...
40 views

### Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can ...
52 views

### How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR. The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254. Is using Fleiss ...
29 views

### Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
33 views

### Am I using Fleiss' kappa coefficient correctly?

I have a dataset where 20 subjects rated 16 audio samples according to some perceptual features on 1-9 scale and I am interested to measure agreement between raters. What I do is following: for each ...
71 views

### Inter-rater agreement with different rater pairings

I have data from the following study design: 100 Patients have been examined by 2 raters each. The raters had to assign a score which consists of 7 subscores and each of this subscore consists of ...
41 views

### Is there such a thing as a multidimensional kappa test?

I am involved with designing a new diagnosis method for a routine clinical procedure. I want to compare the agreement between operators to the agreement between methods. Is there such a thing as a ...
208 views

### How do I calculate reliability between an algorithm and multiple human experts?

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
57 views

### Fisher's exact test vs kappa analysis

I was reading a paper where the authors assessed the association between two different diagnostics tests intended to diagnose the same disease and they performed the analysis with Fisher's exact test. ...
59 views

### Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
184 views

### Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
300 views

### Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
173 views

### Validity of grand mean intra-rater Kappa or ICC

For 5 raters, I have calculated intra-rater Cohen's Kappa statistics for the test-retest nominal ratings, intra-rater Kendall's tau-b statistics for the test-retest ordinal ratings, and intra-rater ...
134 views

### Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
88 views

### What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
19 views

### Combining the standard errors of Cohen's Kappa computed for multiple rater pairs

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method. To ...
116 views

### When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
62 views

### Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
45 views

### Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
88 views

### Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
122 views

### When should I prefer kappa over accuracy for binary classification problems?

I'm using the caret package in R that provides two kind of metrics for classification problems - accuracy and kappa. When should ...
16 views

### R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa

In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value ...
8 views

### Inter annotator aggrement for unbalanced datasets - KAPPA statistics

Can you suggest a better Kappa to measure the agreement between two annotators for unbalanced datasets with two classes?
17 views

### Sample size calculation for weighted Kappa

I will highly appreciate if anyone let me know how to calculate sample size for weighted kappa. I do know how to do it for unweighted kappa.
26 views

### Can Cohen's kappa be used for raters with different but related targets?

There are two binary raters: Rater 1 is certified to assess the ability of people to drive a taxi. Rater 2 is certified to assess the ability of people to drive a lorry. Rater 1 and 2 have both ...
16 views

### small p and small kappa

I have a dataset of roughly 12,000 points. They are boolean values (1 or 0) for a "program" and "outcome". I am trying to find if the program has an impact on outcome. I did a logistic regression in ...
47 views

### Explaining kappa statistic as a percentage

Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa. ...
20 views

### kappa, ICC, and combined score

I am running into a problem in trying to determine whether results from an xray matches results from an ultrasound. I have about 100 patients, and for each patient we measure a, b, c, and d using both ...
46 views

### Kappas calculated using different R packages (epibasix vs. irr); weighted & unweighted kappa

I'm having some issues calculating inter-rater agreement in R. I need to calculate Cohen's kappas (quadratic weighted and unweighted). I've used the epibasix package to calculate unweighted kappas and ...
20 views

### How do I calculate the inter-rater agreement for sequence alignments?

I am having this task of aligning one sequence of sentences to another sequence of sentences. Let's say we have sequence 1 and sequence 2 as an example of the task: Sequence 1:  A B C D E  F ...
216 views

### Difference between ROC and Kappa and TSS

I am using species distribution model to model the absence-presence of species against a set of predictors. I was wondering if anyone can guide me with the following: My model gives me a high ROC ...
25 views

### Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...