Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How to calculate sample size estimation for fleiss kappa?

We have six categories that raters have to choose from. We have an intervention that should improve the rating quality. In previous studies the interobserver agreement is fleiss kappa 0.43 We expect ...
0
votes
0answers
9 views

Determining whether a rating scale is ordinal or nominal from data?

I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g. Very positive. Somewhat positive Neutral Somewhat negative Very ...
0
votes
0answers
6 views

Help Performing Inter-Rater(Coder) Reliability in NVivo

I'm a student using the software NVivo 12 for Windows to perform qualitative coding on interview transcripts. One other person will be coding the same interviews independently so that we can perform ...
0
votes
0answers
10 views

0/1s in the diagonal of weight matrix when computing the weighted kappa

I am trying to compute the weighted kappa using quadratic weights (aka Fleiss-Cohen), but am getting extremely confused with the weights. I am able to compute these in R with no problem when using ...
0
votes
2answers
28 views

Inter-rater agreement after rater 1-based sampling

Rater 1 rated (+/-) some cases. Rater 2 gets to rate all the Rater 1 + cases, but only half of the rater 1 - cases. Is there a way to account for this sampling scheme when calculating inter-rater ...
0
votes
1answer
14 views

Different kappa results from kappa2() and CohenKappa()

I am trying to compute a weighted kappa with confidence intervals in R. I am having trouble understanding why the two functions, DescTools::CohenKappa and irr::kappa2, give different outputs. I need ...
1
vote
1answer
65 views

Calculating the Standard Error and Confidence Interval for Cohen's Quadratic Kappa

I need to evaluate the performance of a machine learning application. One of the evaluation metrics chosen is Cohen's Quadratic Kappa. I found this Python tutorial on how to calculate Cohen's ...
0
votes
0answers
10 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
0
votes
0answers
13 views

Multi-class inter-annotator agreement for overlapping time windows of annotations

My work involves a time series data of sensor measurements. I have a tool to annotate this dataset that allows annotators to flexibly add labels to varying time-windows. I have two annotators who have ...
0
votes
0answers
6 views

How to create two random variables with Kappa distribution with different Kurtosis?

I would like to generate two kappa random variables with the same mean, standard deviation, but with different Skewness and Kurtosis. Is there any ways to transform a dataset to increase/decrease the ...
1
vote
1answer
27 views

Cohen's Kappa using (irr) and kappa2() outputs z and p-value with NaN

I used the kappa2 function from the irr-library in R, however it returned an invalid z- and p-value and a kappa value of exactly 0. I already looked it up and stumbled upon this article Cohen's ...
0
votes
1answer
64 views

Cohen's Kappa for more than two categories

I have data set of teacher's evaluation rated by 4 different raters. The teacher's are evaluated for 13 different categories for example (Interaction with students, lesson delivery etc). All of the 4 ...
2
votes
1answer
53 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
1
vote
1answer
123 views

Fleiss' Kappa v. Cohen's Kappa v. Cronbach's Alpha

I am working on a project that compares two methods of assessing whether certain topics come up in a therapy session. I am comparing a checklist full of dichotomous (y/n) items completed by an ...
0
votes
0answers
17 views

Cohen Kappa Agreement By Chance

As Cohen's Kappa measure inter-rater agreement between two independent observer, it takes out the agreement by chance from observed agreement because there is always probability of random agreement ...
0
votes
0answers
12 views

Best correlation test for an ordinal / nominal dataset

I have a data set in which I have, among others, two variables that measure the degree of stenosis in an artery. Both variables use the same classification which is ordinal (from 1 to 5, meaning 1 = "...
0
votes
0answers
15 views

sample size advice for matching nominal variables

I have two databases of patients and medications. Each database has the same patients. Both have millions of records. My job is to select a sample from each and confirm that the medications match. ...
2
votes
2answers
165 views

Is there a strict relation between Accuracy and Cohen's Kappa (measures of classification quality/agreement)?

Say I have two models (classifiers), $M_1$ and $M_2$, each with their own accuracy w.r.t. the ground truth, and I calculate also Cohen's Kappa between each model and the ground truth (as a measure of ...
0
votes
1answer
27 views

Nonparametric bootstrap to construct confidence intervals for Cohen's Kappa Coefficient

I was recently asked to perform nonparametric bootstrap to construct 95% confidence intervals for $\kappa$ using normal approximation, but I'm not sure how to do this. The only data I was given was ...
4
votes
1answer
719 views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
0
votes
0answers
15 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
0
votes
0answers
11 views

What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...
0
votes
0answers
21 views

Spearman's or weighted kappa?

I am hoping to do a study comparing therapist estimates of an ordinal measure (questionnaire a patient completes) with the physiotherapist's own estimates of the same thing (so the therapist will be ...
2
votes
2answers
105 views

Find if there is a correlation between two paired data points

I have two lists of data with each 10 values from 10 participants. Let’s say the first list contains IQ’s of the 10 participants. And the second list contains how many words each 10 participants can ...
0
votes
0answers
27 views

How do I calculate Inter-rater reliability between one rater and a group of raters?

So let's say I have a study in which I give a categorical rating to bunch of items. Each item could be a pair of images and I have to rate which one is brighter for example. Perhaps three ratings ...
0
votes
0answers
11 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
1
vote
1answer
34 views

weighted cohens kappa categories

I need to ask about how many categories can be included in a weighted kappa or cohens kappa statistic for inter observer reliability. Effectively we need to be reliable in assessing whether the ...
0
votes
0answers
19 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
1
vote
1answer
39 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
0
votes
0answers
14 views

Fleiss' kappa: why the proportion of agreement by chance is calculated with the observered proportions of the categories?

The formula for calculating Fleiss' kappa is: $$ \kappa = \frac{p_o - p_a}{1 - p_a} $$ The aggreement by change ratio is calculated via the sum of squares of observed category proportions: $$ p_a = ...
0
votes
0answers
14 views

Measuring Agreement between more than two coders?

I've to find inter-Coder Agreement for my data. I have around 18 coders to fill a survey for me. Now i'm confused how to find Kappa Statistic value when we have to find agreement between more than two ...
0
votes
0answers
329 views

Fleiss' Kappa: How to interpret p value > 0.05 when Kappa is < 0?

I have 3 raters who have rated 27 items on a Likert scale from 1 to 5 and want to compute Fleiss' Kappa; I do understand that a Kappa below 0 means "poor agreement" but I struggle whether this is ...
1
vote
1answer
30 views

Quadratic weighted kappa error

I have recently downloaded the extension for weighted kappa on SPSS. I followed the procedure of this video. However, at the end I have the following error "some ratings are not integers". Do you have ...
1
vote
2answers
105 views

Is it valid to report the mean of Cohen's kappa coefficients and the associated confidence intervals?

I am studying the interrater and intrarater reliability of a classification system (seven categories; nominal). I recruited 16 unique raters to classify 35 items according to the system. The raters ...
2
votes
1answer
173 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
1answer
93 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
1
vote
1answer
470 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
0
votes
1answer
32 views

Cohen's kappa for agreements between two inter rater metrics?

I have two metrics for measuring source code cohesion. To verify that both have some agreement can I use Cohen kappa for that? Both the metrics have values in range [0 to 1]. It's not necessary if ...
3
votes
1answer
544 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
2
votes
0answers
39 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
1
vote
0answers
21 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
1answer
84 views

Cohen's Kappa in R versus SPSS

I was working with R and SPSS trying to determine Cohen's Kappa for a dataset. I decided to test the two and see if they would end up getting the same numbers. However, they did not. I was wondering ...
1
vote
0answers
14 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
3
votes
1answer
30 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
0answers
57 views

Cohen's Kappa and prevalence and bias in R

I am working on a dataset where two raters indicated whether or not particular features were present in the same individuals. So each individual was assessed by each of these two raters and received ...
0
votes
0answers
36 views

Is Cohen Kappa (Weighted or not) appropriate for medical research? | Agreement between measures

I'm working on a experiment when we test the effectiveness of a new diagnostic compared to an existing measure. Our results will be the format ...
1
vote
1answer
192 views

Can I compute an F1 score when the test data has no examples of one class?

I am working on a 3-class classification problem. We are cross-validating via a Leave-One-Out Approach, and there are some instances where the test data has no instances of one of my three classes. ...
0
votes
0answers
936 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
0
votes
0answers
368 views

How to deal with a negative Kappa in classification?

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows. Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for ...
1
vote
1answer
65 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...