Questions tagged [cohens-kappa]
A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.
190
questions
0
votes
0
answers
8
views
Calculating Cohen's Kappa for a dataset when categories differ between examples
Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
1
vote
1
answer
13
views
Chance Agreement for Fleiss Kappa
In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as
where $p_j$ is defined as the proportion of ratings which were to ...
- 31
0
votes
0
answers
13
views
Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]
I an assessing the agreement between two measurements methods on a binary presence/absence metric.
Here is one example of a contingency table:
...
- 197
0
votes
0
answers
36
views
An overall agreement coefficient for each subject
There are two paired data sets. Each suject has 80 variables gathered from CT scan and the same 80 variables gathered from MRI. How to obtain a global agreement coefficient (Inter-rater reliability) ...
0
votes
0
answers
18
views
How to interpret Fleiss' kappa?
How do you interpret Fleiss' kappa? Can you compare Fleiss' kappa values when the amount of raters differ? For example, Fleiss' kappa with three raters versus a value obtained with more/less raters.
0
votes
0
answers
32
views
How to do sample size calculation for Krippendorff's alpha (paired nominal/categorized data with 10 raters/observers)?
I'm planning an prospective observational trial studying the agreement of 10 doctors (from the same field) in diagnosing certain findings on ultrasound (nominal yes/no outcomes). There will inevitably ...
- 11
1
vote
1
answer
56
views
inter-rater reliability for numeric variables
I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
- 601
1
vote
1
answer
30
views
Inter-rater reliability (complex survey )
Help calculating Inter-rater reliability in a complex online survey (free text).
The questions are free text. We have now coded the potential answers. Examples:
1.) Hobbies (Time in nature, mind-body ...
- 11
0
votes
0
answers
19
views
Why we cannot calculate an ROC curve in cost sensitive learning?
In the Applied Predictive Modeling book, cost sensitivity learning approach, the author(s) write:
One consequence of this approach is that class probabilities cannot be
generated for the model, at ...
0
votes
0
answers
29
views
cohen's kappa power analysis for dependent data
I'm using weighted Cohen's Kappa for calculating the inter-reader agreement between two readers. Data is a medical image of knee of 30 patients. But each knee image we divided in two halves to ...
0
votes
0
answers
104
views
Inter-rater reliability for one subject and multiple classes of categorical data
I am tasked with interpreting the IRR for each question of a recent experiment. For each question, 10 unique individuals observed an image and assigned it to one of four categorical classes. The ...
- 101
1
vote
0
answers
63
views
Can I compare Cohen's and Fleiss' kappa values?
I am running an experiment where a number of human specialists perform a task by hand, and then a computer algorithm performs the same task. I want to determine how similar the algorithm's results are ...
- 111
0
votes
0
answers
59
views
Inter-rater reliability for 2 coders of 30 responses each for 100 different items
I have a data set with the structure outlined below which was coded by two independent raters and I want to determine inter-rater reliability.
The dataset contains responses to 100 different items by ...
- 1
0
votes
1
answer
43
views
What metrics work well in unbalanced assemblies?
I wanted to know if there are some metrics that work well when working with an unbalanced dataset. I know that accuracy is a very bad metric when evaluating a classifier when the data is unbalanced ...
- 135
1
vote
2
answers
172
views
Calculating Cohen's Kappa in SPSS for a systematic review
I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are ...
- 13
0
votes
1
answer
132
views
When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]
I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not ...
- 3
0
votes
1
answer
74
views
Inter Annotator Agreement for Question Answering
I would like to run an Inter Annotator Agreement (IAA) test for Question Answering. I've tried to look for a method to do it, but I wasn't able to get exactly what I need. I've read that there are ...
- 101
0
votes
0
answers
106
views
How to test whether there is a statistically significant difference in agreement(Cohen's kappa) between 2 or more independent groups?
I have 2 methods to compare by difference in race. For example I want to test the agreement between Method 1 and Method 2 between Asians, Americans and Africans.
I know how to calculate kappa ...
- 1
1
vote
0
answers
156
views
How to compare two Kappa values based on the same sample but different methods (two new methods A and B, and a gold standard method C)
Now, I have two assessment methods A and B (both of them are new methods) for assessing diabetes. Moreover, the method C is the gold standard.
Firstly, I ask a rater (Raters X) to use Method A and C (...
0
votes
0
answers
14
views
Proper metric to compare continuous to discrete data? [duplicate]
I have two measurements of the same thing. The first is a subjective rating score, the second is a quantitative metric. The rating score is discrete (0-10), while the quantitative metric is any value ...
- 63
0
votes
0
answers
124
views
Can Fleiss' Kappa test the agreement of three different parameters?
I have a question about statistics. I have a question about the kappa coefficient. I think this coefficient is used for rater reliability.
However, is it possible to test the consistency change of two ...
0
votes
0
answers
94
views
Calculate the mean and the variance of the Kappa statistic
The Kappa statistic was defined as following:
$\kappa = \displaystyle\frac{p_o-p_e}{1-p_e}$, where $ p_o = \displaystyle\sum_{i=1}^{k} p_{ii}$ is the observed agreement, and $p_c = \displaystyle\sum_{...
- 101
1
vote
0
answers
31
views
Calculating the quality of text parsing script
I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research.
The ...
- 11
1
vote
1
answer
101
views
Weighted Kappa with rating variables that have different set of categories
I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT
Rater1 rated the the ...
- 517
0
votes
0
answers
123
views
How can I tell if my classifier is significantly better than chance?
I have a fairly rubbish classifier for a task assigning samples to one of the classes {i, e, l, g}. I would like to know whether it's actually better than random.
All the discussion that I can find is ...
1
vote
0
answers
141
views
How to assess inter-rater reliability when you have multiple observations (and thus ratings) per participant?
I am trying to calculate inter-rater reliability for my data. I have multiple participants and multiple responses per participant. Each participant provided a list of responses, which will then be ...
- 11
0
votes
0
answers
59
views
All Good But Low Kappa
In my experiment, everything looks fine except the Kappa because the Accuracy is very close to the Random Accuracy.
For example,
True Positive = 70689
False Negative = 16098
False Positive = 55
True ...
- 1
3
votes
0
answers
117
views
Accuracy assessment for multinomial probabilities
I am working on a classification problem where the input data are multinational proportions for each point, with $m$>2 groups. And the outcome is likewise a multinomial proportion for that point. ...
- 231
0
votes
0
answers
88
views
Kappa coefficient for variable number of raters
I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories.
...
- 1
1
vote
0
answers
413
views
How to get Kappa and MCC from nested cross-validation?
Let's take this well known example. How could I get the Kappa score and the Matthews correlation coefficient from nested cross-validation?
I've tried to make it with ...
- 23
0
votes
1
answer
405
views
Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?
Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when ...
- 21
0
votes
1
answer
179
views
Inter-rater reliability for multiple raters for one item (subject)
I have data with 3 raters, each of whom rated 3 items as follows:
item a
item b
item c
rater 1
a1
b1
c1
rater 2
a2
b2
c2
rater 3
a3
b3
c3
I successfully calculated the inter-rater reliability for ...
1
vote
1
answer
58
views
Calculating inter-rater reliability with 1 primary rater and a couple other secondary raters
I'm trying to figure out what to use to calculate inter-rater reliability for a set of behavioral coding. There's one individual codes the whole set (primary coder; e.g. codes data ID 1-300) and a few ...
- 11
0
votes
1
answer
79
views
Rotating raters and kappa - appropriate statistic?
I have a several hundred x-rays. Each case is reviewed by 2 of 5 raters and assigned a categorical severity (Mild/Moderate/Severe). So similar data I expect might look like:
...
- 3
0
votes
0
answers
21
views
How to select dichotomous variables?
I have a data set of 57 dichotomous items that constitute a scale for measuring a particular condition. I also have another dichotomous variable of the condition itself. I would like to do some factor ...
- 2,720
0
votes
1
answer
614
views
Inter annotator agreement (or disagreement) for highly imbalance annotations?
I have a time-series dataset which is annotated by 4 individual annotators where I have following things in annotations.
Its possible that one annotator has not annotated all samples (i.e. missing ...
- 1
1
vote
0
answers
105
views
How to determine agreement between two sets?
I have two sets of Ids both drawn from the same background. I'd like to know the appropriate way to compare their similarity. Set A and B are often significantly different in length
eg.
...
- 11
0
votes
1
answer
644
views
Why doesn't it make sense to measure agreement with Chi square?
Why is it more useful to calculate Cohen's Kappa instead of the Pearson Chi Square test to measure the interrater agreement? Could someone please give me an example?
- 35
1
vote
2
answers
852
views
Testing the significant difference between 4 kappa values in one test
This question is about comparing the significant difference between accuracy metrics (that can be derived from a confusion matrix) calculated for four different models.
I have four unsupervised change ...
- 113
3
votes
1
answer
489
views
Can pearson's correlation coefficient be converted to Cohen's kappa?
I'm interested in meta-analysing the agreement between two measures. Depending on whether the variables are continuous/categorical/binary, different studies present agreement as either:
A Pearson's ...
- 331
0
votes
0
answers
198
views
Accuracy, F-Score, and Kappa Together
What can accuracy, f-score, and kappa indicate together for a confusion matrix that each individually can't?
I get F-Score and Accuracy measure to be quite close so I feel it should be sufficient for ...
- 1
1
vote
1
answer
92
views
Hypothesis test on kappa
Problem:
I am an electrical engineer and not a pro statistician.
I have a classification problem and want to compare two classifiers using Cohence kappa metric.
To calculate the kappa value, the ...
- 11
0
votes
0
answers
101
views
Which Kappa to use in R to calculate agreement for binomial data (0 or 1) between two raters?
I have a dataset in which two raters (Human raters VS. ML model) assign "0" to pseudowords if they consider it as being masculine or "1" if they think they are feminine. So, it is ...
2
votes
0
answers
40
views
Inter rater agreement where categories are not predetermined
How can I measure inter-rater agreement, e.g., using the Kappa coefficient, during an exploratory study where the categories are not predetermined?
- 701
2
votes
1
answer
365
views
Inclusion of a NA category in inter-rater agreement calculation
I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not ...
- 85
3
votes
2
answers
985
views
Significant Kappa coefficient with 95% CI spanning 0?
I'm using R's kappa2 function (irr package) to obtain Cohen's Kappa and the cohen.kappa ...
- 31
3
votes
0
answers
33
views
Estimating Irreducible error from kappa statistics for medical applications
In many medical applications that involve some form of classification the data is annotated by expert humans (e.g. histopathologists, radiologists). This is then taken as the ground-truth. However, it ...
- 559
1
vote
0
answers
191
views
How to compare consistency between clustering results and list of values with different levels in R?
I found similar subjects on the website but I may have missed the relation with my own question. I'v seen questions about comparison of clustering results, but here it's more about comparing two lists ...
- 11
1
vote
1
answer
266
views
Test-Retest reliability with multiple raters on different subjects at different times
I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5.
For example, the below data relates to 3 different "Rater"'s ...
- 133
1
vote
0
answers
72
views
Determining whether a rating scale is ordinal or nominal from data?
I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g.
Very positive.
Somewhat positive
Neutral
Somewhat negative
Very ...
- 15.4k