Skip to main content

Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

Is it possible to calculate inter-rater reliability for one item rated by multiple raters with weights?

I have a survey with a number of statements that had participants categorizing the statements into one of the 4 options they were provided. The participants were then asked to rate the confidence of ...
Jay Jakka's user avatar
3 votes
1 answer
49 views

Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
nahp's user avatar
  • 151
1 vote
0 answers
14 views

Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
Anıl's user avatar
  • 11
4 votes
3 answers
527 views

How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
Kevin's user avatar
  • 49
0 votes
0 answers
22 views

How to test similarity between multiple medical differential diagnoses?

I have multiple observations collected by doctors where they give the differential diagnoses, the 10 most probable diseases (d). I want to calculate the inter-observer similarity. my data looks like ...
Timothy's user avatar
1 vote
0 answers
25 views

Analysing agreement with many dimensions to the data

I am tasked with running a sample size calculation and also statistical analysis of a study looking at the agreement between two 'tools'. This is a binary outcome so looking mainly at the different ...
Dean Langan's user avatar
1 vote
0 answers
45 views

Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
user21584544's user avatar
3 votes
0 answers
51 views

Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
Yip's user avatar
  • 31
0 votes
1 answer
30 views

Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
Novra's user avatar
  • 1
0 votes
0 answers
52 views

Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
Lukas Wallrich's user avatar
0 votes
0 answers
132 views

Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
Nelson Liu's user avatar
1 vote
1 answer
105 views

Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
crabnebul's user avatar
1 vote
0 answers
51 views

Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
Timelate's user avatar
  • 309
2 votes
1 answer
422 views

inter-rater reliability for numeric variables

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
Larissa Cury's user avatar
1 vote
1 answer
41 views

Inter-rater reliability (complex survey )

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body ...
Vit Frac's user avatar
1 vote
0 answers
139 views

Can I compare Cohen's and Fleiss' kappa values?

I am running an experiment where a number of human specialists perform a task by hand, and then a computer algorithm performs the same task. I want to determine how similar the algorithm's results are ...
CesarGon's user avatar
  • 111
0 votes
1 answer
59 views

What metrics work well in unbalanced assemblies?

I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
PicaPython's user avatar
1 vote
2 answers
530 views

Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
Iris's user avatar
  • 13
1 vote
1 answer
381 views

When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]

I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not ...
Cameron's user avatar
  • 13
0 votes
1 answer
132 views

Inter Annotator Agreement for Question Answering

I would like to run an Inter Annotator Agreement (IAA) test for Question Answering. I've tried to look for a method to do it, but I wasn't able to get exactly what I need. I've read that there are ...
SilentCloud's user avatar
0 votes
0 answers
177 views

How to test whether there is a statistically significant difference in agreement(Cohen's kappa) between 2 or more independent groups?

I have 2 methods to compare by difference in race. For example I want to test the agreement between Method 1 and Method 2 between Asians, Americans and Africans. I know how to calculate kappa ...
Merry's user avatar
  • 1
1 vote
0 answers
282 views

How to compare two Kappa values based on the same sample but different methods (two new methods A and B, and a gold standard method C)

Now, I have two assessment methods A and B (both of them are new methods) for assessing diabetes. Moreover, the method C is the gold standard. Firstly, I ask a rater (Raters X) to use Method A and C (...
skywalker21th's user avatar
0 votes
0 answers
16 views

Proper metric to compare continuous to discrete data? [duplicate]

I have two measurements of the same thing. The first is a subjective rating score, the second is a quantitative metric. The rating score is discrete (0-10), while the quantitative metric is any value ...
user2551700's user avatar
0 votes
0 answers
159 views

Can Fleiss' Kappa test the agreement of three different parameters?

I have a question about statistics. I have a question about the kappa coefficient. I think this coefficient is used for rater reliability. However, is it possible to test the consistency change of two ...
user343047's user avatar
1 vote
0 answers
183 views

Calculate the mean and the variance of the Kappa statistic

The Kappa statistic was defined as following: $\kappa = \displaystyle\frac{p_o-p_e}{1-p_e}$, where $ p_o = \displaystyle\sum_{i=1}^{k} p_{ii}$ is the observed agreement, and $p_c = \displaystyle\sum_{...
math112358's user avatar
1 vote
0 answers
35 views

Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
Peter's user avatar
  • 11
1 vote
1 answer
150 views

Weighted Kappa with rating variables that have different set of categories

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT Rater1 rated the the ...
learners's user avatar
  • 579
0 votes
0 answers
226 views

How can I tell if my classifier is significantly better than chance?

I have a fairly rubbish classifier for a task assigning samples to one of the classes {i, e, l, g}. I would like to know whether it's actually better than random. All the discussion that I can find is ...
Allan from Wildboarclough's user avatar
1 vote
0 answers
210 views

How to assess inter-rater reliability when you have multiple observations (and thus ratings) per participant?

I am trying to calculate inter-rater reliability for my data. I have multiple participants and multiple responses per participant. Each participant provided a list of responses, which will then be ...
milla1606's user avatar
0 votes
0 answers
153 views

All Good But Low Kappa

In my experiment, everything looks fine except the Kappa because the Accuracy is very close to the Random Accuracy. For example, True Positive = 70689 False Negative = 16098 False Positive = 55 True ...
GISEnthu's user avatar
4 votes
1 answer
163 views

Accuracy assessment for multinomial probabilities

I am working on a classification problem where the input data are multinational proportions for each point, with $m$>2 groups. And the outcome is likewise a multinomial proportion for that point. ...
user3037237's user avatar
1 vote
0 answers
149 views

Kappa coefficient for variable number of raters

I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories. ...
NST's user avatar
  • 11
1 vote
0 answers
607 views

How to get Kappa and MCC from nested cross-validation?

Let's take this well known example. How could I get the Kappa score and the Matthews correlation coefficient from nested cross-validation? I've tried to make it with ...
István Szabó's user avatar
2 votes
1 answer
793 views

Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when ...
Emily Y's user avatar
  • 51
0 votes
1 answer
376 views

Inter-rater reliability for multiple raters for one item (subject)

I have data with 3 raters, each of whom rated 3 items as follows: item a item b item c rater 1 a1 b1 c1 rater 2 a2 b2 c2 rater 3 a3 b3 c3 I successfully calculated the inter-rater reliability for ...
tobedeclared's user avatar
1 vote
1 answer
73 views

Calculating inter-rater reliability with 1 primary rater and a couple other secondary raters

I'm trying to figure out what to use to calculate inter-rater reliability for a set of behavioral coding. There's one individual codes the whole set (primary coder; e.g. codes data ID 1-300) and a few ...
user319810's user avatar
1 vote
1 answer
130 views

Rotating raters and kappa - appropriate statistic?

I have a several hundred x-rays. Each case is reviewed by 2 of 5 raters and assigned a categorical severity (Mild/Moderate/Severe). So similar data I expect might look like: ...
anbk's user avatar
  • 13
0 votes
0 answers
30 views

How to select dichotomous variables?

I have a data set of 57 dichotomous items that constitute a scale for measuring a particular condition. I also have another dichotomous variable of the condition itself. I would like to do some factor ...
JonB's user avatar
  • 2,880
0 votes
1 answer
1k views

Inter annotator agreement (or disagreement) for highly imbalance annotations?

I have a time-series dataset which is annotated by 4 individual annotators where I have following things in annotations. Its possible that one annotator has not annotated all samples (i.e. missing ...
manv's user avatar
  • 1
1 vote
0 answers
117 views

How to determine agreement between two sets?

I have two sets of Ids both drawn from the same background. I'd like to know the appropriate way to compare their similarity. Set A and B are often significantly different in length eg. ...
Mike's user avatar
  • 11
0 votes
1 answer
1k views

Why doesn't it make sense to measure agreement with Chi square?

Why is it more useful to calculate Cohen's Kappa instead of the Pearson Chi Square test to measure the interrater agreement? Could someone please give me an example?
Check's user avatar
  • 35
1 vote
2 answers
1k views

Testing the significant difference between 4 kappa values in one test

This question is about comparing the significant difference between accuracy metrics (that can be derived from a confusion matrix) calculated for four different models. I have four unsupervised change ...
saQuist's user avatar
  • 113
4 votes
1 answer
872 views

Can pearson's correlation coefficient be converted to Cohen's kappa?

I'm interested in meta-analysing the agreement between two measures. Depending on whether the variables are continuous/categorical/binary, different studies present agreement as either: A Pearson's ...
JRB's user avatar
  • 471
0 votes
0 answers
352 views

Accuracy, F-Score, and Kappa Together

What can accuracy, f-score, and kappa indicate together for a confusion matrix that each individually can't? I get F-Score and Accuracy measure to be quite close so I feel it should be sufficient for ...
GISEnthu's user avatar
1 vote
1 answer
182 views

Hypothesis test on kappa

Problem: I am an electrical engineer and not a pro statistician. I have a classification problem and want to compare two classifiers using Cohence kappa metric. To calculate the kappa value, the ...
Nina's user avatar
  • 11
0 votes
0 answers
120 views

Which Kappa to use in R to calculate agreement for binomial data (0 or 1) between two raters?

I have a dataset in which two raters (Human raters VS. ML model) assign "0" to pseudowords if they consider it as being masculine or "1" if they think they are feminine. So, it is ...
Natália Resende's user avatar
2 votes
0 answers
48 views

Inter rater agreement where categories are not predetermined

How can I measure inter-rater agreement, e.g., using the Kappa coefficient, during an exploratory study where the categories are not predetermined?
khatchad's user avatar
  • 761
3 votes
1 answer
735 views

Inclusion of a NA category in inter-rater agreement calculation

I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not ...
Dom42's user avatar
  • 115
3 votes
2 answers
2k views

Significant Kappa coefficient with 95% CI spanning 0?

I'm using R's kappa2 function (irr package) to obtain Cohen's Kappa and the cohen.kappa ...
user302969's user avatar
3 votes
0 answers
39 views

Estimating Irreducible error from kappa statistics for medical applications

In many medical applications that involve some form of classification the data is annotated by expert humans (e.g. histopathologists, radiologists). This is then taken as the ground-truth. However, it ...
N Blake's user avatar
  • 579