# Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

193 questions
Filter by
Sorted by
Tagged with
13 views

### Is it possible to calculate inter-rater reliability for one item rated by multiple raters with weights?

I have a survey with a number of statements that had participants categorizing the statements into one of the 4 options they were provided. The participants were then asked to rate the confidence of ...
49 views

### Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
• 151
1 vote
14 views

### Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
• 11
527 views

### How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
• 49
22 views

### How to test similarity between multiple medical differential diagnoses?

I have multiple observations collected by doctors where they give the differential diagnoses, the 10 most probable diseases (d). I want to calculate the inter-observer similarity. my data looks like ...
1 vote
25 views

### Analysing agreement with many dimensions to the data

I am tasked with running a sample size calculation and also statistical analysis of a study looking at the agreement between two 'tools'. This is a binary outcome so looking mainly at the different ...
1 vote
45 views

### Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
51 views

### Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
• 31
30 views

### Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
52 views

### Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
132 views

### Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
1 vote
105 views

### Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
1 vote
51 views

### Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
• 309
422 views

### inter-rater reliability for numeric variables

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
• 781
1 vote
41 views

### Inter-rater reliability (complex survey )

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body ...
• 11
1 vote
139 views

### Can I compare Cohen's and Fleiss' kappa values?

I am running an experiment where a number of human specialists perform a task by hand, and then a computer algorithm performs the same task. I want to determine how similar the algorithm's results are ...
• 111
59 views

### What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
• 157
1 vote
530 views

### Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
• 13
1 vote
381 views

### When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]

I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not ...
• 13
132 views

### Inter Annotator Agreement for Question Answering

I would like to run an Inter Annotator Agreement (IAA) test for Question Answering. I've tried to look for a method to do it, but I wasn't able to get exactly what I need. I've read that there are ...
• 101
177 views

### How to test whether there is a statistically significant difference in agreement(Cohen's kappa) between 2 or more independent groups?

I have 2 methods to compare by difference in race. For example I want to test the agreement between Method 1 and Method 2 between Asians, Americans and Africans. I know how to calculate kappa ...
1 vote
282 views

### How to compare two Kappa values based on the same sample but different methods (two new methods A and B, and a gold standard method C)

Now, I have two assessment methods A and B (both of them are new methods) for assessing diabetes. Moreover, the method C is the gold standard. Firstly, I ask a rater (Raters X) to use Method A and C (...
16 views

### Proper metric to compare continuous to discrete data? [duplicate]

I have two measurements of the same thing. The first is a subjective rating score, the second is a quantitative metric. The rating score is discrete (0-10), while the quantitative metric is any value ...
• 135
159 views

### Can Fleiss' Kappa test the agreement of three different parameters?

I have a question about statistics. I have a question about the kappa coefficient. I think this coefficient is used for rater reliability. However, is it possible to test the consistency change of two ...
1 vote
183 views

• 115