Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

IRR Cohen's Kappa Shiny R

I'm needing to make a shiny app to calculate Cohen's Kappa for a series of different circumstances in the context of interviewers ratings of an interviewee, so after the interview the managers can ...
0
votes
0answers
14 views

What measure of agreeement should I use for time series data?

I am trying to compare the agreement between two measurements. Each measurement tells me when a device is worn or not worn (binary) and is sampled at the same frequency (1Hz). The measurements were ...
1
vote
0answers
19 views

Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
1
vote
1answer
41 views

Weighted Kappa with rating variables that have different set of categories

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT Rater1 rated the the ...
0
votes
0answers
9 views

Assessing Consistency and Agreement of Two Groups of Raters in an ordinal dataset

I'm using a real dataset of convenience to look at one group of subjects who are being assessed for "frailty". The goal is to see whether assessments by non-experts are comparable to experts....
0
votes
0answers
34 views

How can I tell if my classifier is significantly better than chance?

I have a fairly rubbish classifier for a task assigning samples to one of the classes {i, e, l, g}. I would like to know whether it's actually better than random. All the discussion that I can find is ...
1
vote
0answers
27 views

How to assess inter-rater reliability when you have multiple observations (and thus ratings) per participant?

I am trying to calculate inter-rater reliability for my data. I have multiple participants and multiple responses per participant. Each participant provided a list of responses, which will then be ...
0
votes
0answers
27 views

interrater reliability alternatives to kappa

I have applied a binary classifier to a raster. Within the raster, I have "ground" truth plots where pixels have been manually assigned to one of the two classes. I would like to assess the ...
0
votes
0answers
25 views

All Good But Low Kappa

In my experiment, everything looks fine except the Kappa because the Accuracy is very close to the Random Accuracy. For example, True Positive = 70689 False Negative = 16098 False Positive = 55 True ...
3
votes
0answers
29 views

Accuracy assessment for multinomial probabilities

I am working on a classification problem where the input data are multinational proportions for each point, with $m$>2 groups. And the outcome is likewise a multinomial proportion for that point. ...
0
votes
0answers
49 views

Kappa coefficient for variable number of raters

I want to calculate the Inter-rater Reliability of an annotated dataset, where the number of raters for each subject is variable (3-5 raters) and subjects can be annotated with one of 7 categories. ...
1
vote
0answers
124 views

How to get Kappa and MCC from nested cross-validation?

Let's take this well known example. How could I get the Kappa score and the Matthews correlation coefficient from nested cross-validation? I've tried to make it with ...
0
votes
1answer
51 views

Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when ...
0
votes
1answer
30 views

Inter-rater reliability for multiple raters for one item (subject)

I have data with 3 raters, each of whom rated 3 items as follows: item a item b item c rater 1 a1 b1 c1 rater 2 a2 b2 c2 rater 3 a3 b3 c3 I successfully calculated the inter-rater reliability for ...
1
vote
1answer
16 views

Calculating inter-rater reliability with 1 primary rater and a couple other secondary raters

I'm trying to figure out what to use to calculate inter-rater reliability for a set of behavioral coding. There's one individual codes the whole set (primary coder; e.g. codes data ID 1-300) and a few ...
0
votes
1answer
12 views

Rotating raters and kappa - appropriate statistic?

I have a several hundred x-rays. Each case is reviewed by 2 of 5 raters and assigned a categorical severity (Mild/Moderate/Severe). So similar data I expect might look like: ...
0
votes
0answers
11 views

How to select dichotomous variables?

I have a data set of 57 dichotomous items that constitute a scale for measuring a particular condition. I also have another dichotomous variable of the condition itself. I would like to do some factor ...
0
votes
1answer
145 views

Inter annotator agreement (or disagreement) for highly imbalance annotations?

I have a time-series dataset which is annotated by 4 individual annotators where I have following things in annotations. Its possible that one annotator has not annotated all samples (i.e. missing ...
1
vote
0answers
57 views

How to determine agreement between two sets?

I have two sets of Ids both drawn from the same background. I'd like to know the appropriate way to compare their similarity. Set A and B are often significantly different in length eg. ...
0
votes
0answers
24 views

Combining two independent annotator agreements

We have 4 annotators that are going to manually classify 500 text data, and we want to compute the inter-annotator agreement scores. We think that the best way to do it is just to ask each of the four ...
0
votes
0answers
13 views

Cohens Kappa for multiple categories

I have a data set of two observers evaluating 50 patient xrays. We are evaluating hip replacement femoral stem loosening by gruen zone, meaning a single patient may have more than one zone of ...
0
votes
1answer
54 views

Why doesn't it make sense to measure agreement with Chi square?

Why is it more useful to calculate Cohen's Kappa instead of the Pearson Chi Square test to measure the interrater agreement? Could someone please give me an example?
1
vote
2answers
256 views

Testing the significant difference between 4 kappa values in one test

This question is about comparing the significant difference between accuracy metrics (that can be derived from a confusion matrix) calculated for four different models. I have four unsupervised change ...
2
votes
1answer
143 views

Can pearson's correlation coefficient be converted to Cohen's kappa?

I'm interested in meta-analysing the agreement between two measures. Depending on whether the variables are continuous/categorical/binary, different studies present agreement as either: A Pearson's ...
0
votes
0answers
60 views

Accuracy, F-Score, and Kappa Together

What can accuracy, f-score, and kappa indicate together for a confusion matrix that each individually can't? I get F-Score and Accuracy measure to be quite close so I feel it should be sufficient for ...
1
vote
1answer
49 views

Hypothesis test on kappa

Problem: I am an electrical engineer and not a pro statistician. I have a classification problem and want to compare two classifiers using Cohence kappa metric. To calculate the kappa value, the ...
0
votes
0answers
48 views

Which Kappa to use in R to calculate agreement for binomial data (0 or 1) between two raters?

I have a dataset in which two raters (Human raters VS. ML model) assign "0" to pseudowords if they consider it as being masculine or "1" if they think they are feminine. So, it is ...
1
vote
0answers
21 views

Inter rater agreement where categories are not predetermined

How can I measure inter-rater agreement, e.g., using the Kappa coefficient, during an exploratory study where the categories are not predetermined?
2
votes
1answer
112 views

Inclusion of a NA category in inter-rater agreement calculation

I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not ...
3
votes
2answers
333 views

Significant Kappa coefficient with 95% CI spanning 0?

I'm using R's kappa2 function (irr package) to obtain Cohen's Kappa and the cohen.kappa ...
3
votes
0answers
29 views

Estimating Irreducible error from kappa statistics for medical applications

In many medical applications that involve some form of classification the data is annotated by expert humans (e.g. histopathologists, radiologists). This is then taken as the ground-truth. However, it ...
1
vote
0answers
87 views

How to compare consistency between clustering results and list of values with different levels in R?

I found similar subjects on the website but I may have missed the relation with my own question. I'v seen questions about comparison of clustering results, but here it's more about comparing two lists ...
1
vote
0answers
100 views

Test-Retest reliability with multiple raters on different subjects at different times

I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5. For example, the below data relates to 3 different "Rater"'s ...
1
vote
0answers
31 views

Determining whether a rating scale is ordinal or nominal from data?

I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g. Very positive. Somewhat positive Neutral Somewhat negative Very ...
1
vote
2answers
37 views

Inter-rater agreement after rater 1-based sampling

Rater 1 rated (+/-) some cases. Rater 2 gets to rate all the Rater 1 + cases, but only half of the rater 1 - cases. Is there a way to account for this sampling scheme when calculating inter-rater ...
0
votes
1answer
68 views

Different kappa results from kappa2() and CohenKappa()

I am trying to compute a weighted kappa with confidence intervals in R. I am having trouble understanding why the two functions, DescTools::CohenKappa and irr::kappa2, give different outputs. I need ...
1
vote
1answer
561 views

Calculating the Standard Error and Confidence Interval for Cohen's Quadratic Kappa

I need to evaluate the performance of a machine learning application. One of the evaluation metrics chosen is Cohen's Quadratic Kappa. I found this Python tutorial on how to calculate Cohen's ...
1
vote
0answers
45 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
1
vote
1answer
263 views

Cohen's Kappa using (irr) and kappa2() outputs z and p-value with NaN

I used the kappa2 function from the irr-library in R, however it returned an invalid z- and p-value and a kappa value of exactly 0. I already looked it up and stumbled upon this article Cohen's ...
0
votes
1answer
932 views

Cohen's Kappa for more than two categories

I have data set of teacher's evaluation rated by 4 different raters. The teacher's are evaluated for 13 different categories for example (Interaction with students, lesson delivery etc). All of the 4 ...
2
votes
1answer
469 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
1
vote
1answer
898 views

Fleiss' Kappa v. Cohen's Kappa v. Cronbach's Alpha

I am working on a project that compares two methods of assessing whether certain topics come up in a therapy session. I am comparing a checklist full of dichotomous (y/n) items completed by an ...
0
votes
0answers
31 views

sample size advice for matching nominal variables

I have two databases of patients and medications. Each database has the same patients. Both have millions of records. My job is to select a sample from each and confirm that the medications match. ...
3
votes
2answers
425 views

Is there a strict relation between Accuracy and Cohen's Kappa (measures of classification quality/agreement)?

Say I have two models (classifiers), $M_1$ and $M_2$, each with their own accuracy w.r.t. the ground truth, and I calculate also Cohen's Kappa between each model and the ground truth (as a measure of ...
0
votes
1answer
89 views

Nonparametric bootstrap to construct confidence intervals for Cohen's Kappa Coefficient

I was recently asked to perform nonparametric bootstrap to construct 95% confidence intervals for $\kappa$ using normal approximation, but I'm not sure how to do this. The only data I was given was ...
6
votes
1answer
2k views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
2
votes
2answers
350 views

Find if there is a correlation between two paired data points

I have two lists of data with each 10 values from 10 participants. Let’s say the first list contains IQ’s of the 10 participants. And the second list contains how many words each 10 participants can ...
1
vote
1answer
61 views

weighted cohens kappa categories

I need to ask about how many categories can be included in a weighted kappa or cohens kappa statistic for inter observer reliability. Effectively we need to be reliable in assessing whether the ...
1
vote
1answer
57 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
1
vote
1answer
48 views

Quadratic weighted kappa error

I have recently downloaded the extension for weighted kappa on SPSS. I followed the procedure of this video. However, at the end I have the following error "some ratings are not integers". Do you have ...