Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

2
votes
1answer
19 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
0answers
8 views

Cohen's Kappa and prevalence and bias in R

I am working on a dataset where two raters indicated whether or not particular features were present in the same individuals. So each individual was assessed by each of these two raters and received ...
0
votes
0answers
19 views

Is Cohen Kappa (Weighted or not) appropriate for medical research? | Agreement between measures

I'm working on a experiment when we test the effectiveness of a new diagnostic compared to an existing measure. Our results will be the format ...
0
votes
1answer
16 views

Can I compute an F1 score when the test data has no examples of one class?

I am working on a 3-class classification problem. We are cross-validating via a Leave-One-Out Approach, and there are some instances where the test data has no instances of one of my three classes. ...
0
votes
0answers
69 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
0
votes
0answers
46 views

How to deal with a negative Kappa in classification?

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows. Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for ...
1
vote
0answers
32 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
1
vote
0answers
124 views

Calculation of accuracy (and Cohen's kappa) using sensitivity, specificity, positive and negative predictive values

I read How to calculate specificity from accuracy and sensitivity, but I have two diagnostic performance measures more. Please correct me if I am wrong: if Sensitivity=TP/(TP+FN) Specificity=TN/(TN+...
2
votes
2answers
54 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
1
vote
2answers
125 views

Interpreting SPSS Cohen's Kappa output

When running Cohen's Kappa in SPSS, it outputs a few things. Cohen's Kappa Asymptotic Standard Error (ASE) Approximate T Approximate Significance If I have a Kappa of 0.90 and ASE of .012, can I ...
0
votes
0answers
15 views

In the results of Cohen's Kappa, how do you interpret p-value and p-value = 0? [duplicate]

I am calculating Cohen's Kappa, which give me this result ( It's calculated using the irr package in R): ...
1
vote
1answer
26 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
0
votes
0answers
29 views

What kind of statistical test to perform to show concordance with predicted and observed results?

obs.clock <-c(2,1,1,1,2,1,1,5,1,2,2,7,3,1,1,1) pred.clock <-c(1,1,2,5,2,6,1,1,1,4,2,5,5,1,1,1) I need help with something, which might be very simple. I ...
0
votes
0answers
50 views

Cohen's kappa for repeated measures longitudinal data

I have a dataset with more than 100 repeated measures of observations across multiple subjects. I want to estimate the agreement between two of the variables measured repeatedly. Note that each ...
0
votes
0answers
13 views

Seeking a measure of multiple rater consistency for cateorical ratings with sparse data

I have multiple raters and sparse data. The raters can decide how many attributes apply to a person. I have many empty cells due to many attributes and few attributes actually chosen. What tools ...
0
votes
0answers
44 views

Calculating Kappa correlation coefficients of the data to find agreement between two raters

I am trying to calculate Kappa correlation coefficients of following data to find agreement between two raters. However, from the data set, I can see that the raters are agreeing on almost all the ...
0
votes
0answers
10 views

Measuring effectivenes of raters using Inter rater agreement

I am conducting a general survey where I get results from 5 people. When I measured the inter-rater agreement using Cohen's kappa from 4 people I got kappa value as 0.49 but when I included the 5th ...
1
vote
1answer
220 views

Using Cohen's kappa for multiple item types (e.g. binary and non-binary)

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). ...
0
votes
0answers
482 views

On the difference between Kappa and Accuracy

I have an imbalanced dataset with 5 classes. The distribution is roughly: Class 1: 0.5 Class 2: 0.25 Class 3: 0.15 Class 4: 0.05 Class 5: 0.05 I've been testing some different classification ...
0
votes
0answers
32 views

Kappa Statistic Intuition

Building Intuition | Cohens Kappa Coefficient There are several descent posts addressing this issue, but only one that gets at my main concern; When would one want to use the Kappa Statistic? This ...
1
vote
1answer
37 views

Explanation of the denominator in Cohen's kappa score?

Cohen's kappa with two raters is: $$\kappa = \frac{p_o - p_a}{1 - p_a}$$ where $p_a$ is probability of agreement by chance, and $p_o$ is the observed rate of agreement. I can't figure out why the ...
1
vote
0answers
61 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
63 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
0
votes
0answers
44 views

Inter-Rater Reliability (IRR) Comparison: Is it acceptable to use and average Kappas when comparing rater score to subject scores?

I am currently working on my thesis project and I have hit a bit of a snag. I have two raters who have given participants scores based upon what they wrote in open response questions. After using ...
0
votes
0answers
50 views

What are “z-normalized scores” in the context of Fleiss' kappa coefficient?

I am tasked at work with evaluating inter-rater reliability for an experiment where speakers had to indicate on a print-out of a text which words they hear as prominent and where they hear a boundary (...
0
votes
0answers
42 views

Get kappa for prediction and observation in r

I need to perform kappa for predicted and observed values to see how the agreement is. There are twelve categories; it's sort of like a clock face. There are altogether 16 observations. The idea is, ...
2
votes
1answer
41 views

Relation between Kappa Index of Agreement and Cohen's kappa

I was reading a paper in which Kappa Index of Agreement (KIA) was calculated. I searched the web for a reference but I couldn't find any decent definition. My first result was instead a Wikipedia ...
0
votes
1answer
257 views

Cohen's Kappa: is it valid to average kappa for different rater pairs across multiple trials?

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of ...
-1
votes
0answers
48 views

High correlation but different median scores. How can this be possible?

I am writing a research paper. We asked the same patients to fill out a form at two different times. Data is non-normal. I used Spearman's correlation, which gave me a moderate correlation between ...
0
votes
1answer
143 views

Intercoder reliability without mutually-exclusive codes

I have the results from qualitatively coding the contents of ~100 images, with 7 categories and 96 tags in total. It's incredibly difficult to define mutually exclusive codes for the content we're ...
0
votes
1answer
141 views

NaN produced from Cohen Kappa without 100% agreement

Considering the following data frame called coris_d0: ...
0
votes
1answer
676 views

Fleiss kappa in R giving strange results

I have an experiment where 4 raters gave their responses to 4 stimuli, and I need to calculate the Fleiss Kappa to check the agreements of the raters. However, I get strange results from the R ...
1
vote
0answers
777 views

How exactly is sample size determined in inter-rater reliability study?

I am conducting an inter-rater reliability study where we have four raters and the form of the rating is "Yes vs. No". Correspondingly, we chose to use Kappa value as our outcome measure. Now we want ...
2
votes
1answer
220 views

Cohen's Kappa Classifier vs. Groundtruth

The user guide on Cohen's kappa of the scikit-learn documentation states the following: The function cohen_kappa_score computes Cohen’s kappa statistic. This ...
2
votes
1answer
560 views

How to use kappa() or cohen.kappa() to check for matching assigments between observers

down vote favorite Example 1. In a common situation, if we have the following data: ...
2
votes
2answers
465 views

Cohen's kappa as a classifier “strength” estimator

What are the strengths or weaknesses of using the higher Cohen's kappa score to select a classifier from a pool of candidates? Back-story: I had a pool of candidate classifiers (~200 candidate models)...
0
votes
0answers
30 views

Adjust for rater when examining agreement or Kappa

I am conducting a 10-question survey, with responses of 'yes' and 'no' to each. I want to compare responses to a known 'gold-standard' (also with responses of yes and no). Suppose I have 100 ...
1
vote
2answers
769 views

Facing unbalanced data: AUC vs. Cohen's Kappa vs. Balanced Misclassification Rate

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest". So, I chose two ways ...
1
vote
1answer
237 views

Linear and Quadratic Weighted Kappa SPSS Error

I am using both Linear and Quadratic weighted kappa to examine the extent to which two measures (a quantitative clinical interview, and a corresponding self-report questionnaire) agree at the item ...
3
votes
1answer
3k views

Calculate Kappa statistic in R [closed]

Currently I am trying to calculate Kappa for a confusion matrix. Unfortunately, I never used Kappa in R before. I found this description how to use it: class1 A vector describing a classification of ...
0
votes
0answers
73 views

Test Retest reliability with different rater numbers

I was wondering whether its possible to run a test-retest reliability analysis, but with a different number of raters for the two time points. I have collected personality data and am looking at ...
0
votes
1answer
522 views

Compute Cohen's kappa in multi-label classification

We have 2 different interpretations for our data. Here is an example: ...
2
votes
0answers
180 views

Inter-rater reliability - when Cohen's Kappa doesn't work

I am trying to calculate inter-rater reliability for two raters using a dichotomous scale. In one part of the study, raters have to decide from a list which word candidates of a child's utterances ...
1
vote
1answer
153 views

Calculate sample size for a moderation exercise, using Cohen's Kappa

I need to conduct a moderation study using 3-4 raters, a scale of 5-10 items, and between 1 and 5 samples. Would my Kappa coefficient be meaningful with this sample size? Would I be able to generalise ...
1
vote
2answers
177 views

Von Mises mixtures: What is the maximum kappa value?

I have an R script that is estimating the concentration parameters (kappa) for Von Mises mixtures. This script is giving me values of kappa that are 60, 100 e.c.t. I was expecting kappa to be less ...
1
vote
1answer
1k views

Explain Cohen's kappa in a simplest way? [duplicate]

I'm learning statistics but don't really understand the result of kappa value posted in the first sample here. Is there anyone help me explain in English what the value 0.915 means? (observed ...
3
votes
1answer
1k views

Using Cohen's kappa statistic for evaluating a binary classifier

I am using the caret package to perform predictive modeling on a binary target variable. The outcome is very unbalanced so it is suggested to use the Kappa statistics to evaluate the binary classifier....
1
vote
0answers
31 views

Cohen's Kappa: pairs of raters out of three persons [duplicate]

For example there are 300 targets that need to be rated. There are 3 researchers (X, Y, Z). Each target should be rated by two persons. So I cut the amount of all targets into 3 blocks. Block A: ...
0
votes
1answer
133 views

Calculating agreement between 3 users with continuous data

I have a dataset of consisting of volume measurements as taken by 3 users in two trials. That is to say, each user rated each point twice, therefore I have 6 measurements in total for each point. I ...
2
votes
1answer
497 views

Kappa Statistic for Variable Number of Raters

I have $N$ documents. Each document $N_i$ is annotated by a variable number of raters $n_i \in [3, 10]$ into two categories. Is there a way to compute the Kappa statistic for these annotations? My ...