Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

2
votes
1answer
54 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
0
votes
0answers
14 views

Kappa between two groups of people

So I'm trying to compute Kappa. I've done some research on it and it seems like Cohen's Kappa is generally calculated between two people, say ...
0
votes
0answers
38 views

Difference in insight gained from Cohens' Kappa vs McNemar

I would like to understand the difference in what a test with Cohen's Kappa tells you about your data compared to what McNemar's test tells you. If Cohen's Kappa measures agreement between two ...
2
votes
0answers
12 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
1
vote
0answers
9 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
0answers
24 views

Cohen's Kappa in R versus SPSS

I was working with R and SPSS trying to determine Cohen's Kappa for a dataset. I decided to test the two and see if they would end up getting the same numbers. However, they did not. I was wondering ...
1
vote
0answers
6 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
2
votes
1answer
20 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
0answers
14 views

Cohen's Kappa and prevalence and bias in R

I am working on a dataset where two raters indicated whether or not particular features were present in the same individuals. So each individual was assessed by each of these two raters and received ...
0
votes
0answers
20 views

Is Cohen Kappa (Weighted or not) appropriate for medical research? | Agreement between measures

I'm working on a experiment when we test the effectiveness of a new diagnostic compared to an existing measure. Our results will be the format ...
0
votes
1answer
26 views

Can I compute an F1 score when the test data has no examples of one class?

I am working on a 3-class classification problem. We are cross-validating via a Leave-One-Out Approach, and there are some instances where the test data has no instances of one of my three classes. ...
0
votes
0answers
161 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
0
votes
0answers
107 views

How to deal with a negative Kappa in classification?

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows. Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for ...
1
vote
0answers
37 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
1
vote
0answers
212 views

Calculation of accuracy (and Cohen's kappa) using sensitivity, specificity, positive and negative predictive values

I read How to calculate specificity from accuracy and sensitivity, but I have two diagnostic performance measures more. Please correct me if I am wrong: if Sensitivity=TP/(TP+FN) Specificity=TN/(TN+...
2
votes
2answers
63 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
1
vote
2answers
176 views

Interpreting SPSS Cohen's Kappa output

When running Cohen's Kappa in SPSS, it outputs a few things. Cohen's Kappa Asymptotic Standard Error (ASE) Approximate T Approximate Significance If I have a Kappa of 0.90 and ASE of .012, can I ...
0
votes
0answers
15 views

In the results of Cohen's Kappa, how do you interpret p-value and p-value = 0? [duplicate]

I am calculating Cohen's Kappa, which give me this result ( It's calculated using the irr package in R): ...
1
vote
1answer
32 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
0
votes
0answers
30 views

What kind of statistical test to perform to show concordance with predicted and observed results?

obs.clock <-c(2,1,1,1,2,1,1,5,1,2,2,7,3,1,1,1) pred.clock <-c(1,1,2,5,2,6,1,1,1,4,2,5,5,1,1,1) I need help with something, which might be very simple. I ...
0
votes
0answers
56 views

Cohen's kappa for repeated measures longitudinal data

I have a dataset with more than 100 repeated measures of observations across multiple subjects. I want to estimate the agreement between two of the variables measured repeatedly. Note that each ...
0
votes
0answers
15 views

Seeking a measure of multiple rater consistency for cateorical ratings with sparse data

I have multiple raters and sparse data. The raters can decide how many attributes apply to a person. I have many empty cells due to many attributes and few attributes actually chosen. What tools ...
0
votes
0answers
10 views

Measuring effectivenes of raters using Inter rater agreement

I am conducting a general survey where I get results from 5 people. When I measured the inter-rater agreement using Cohen's kappa from 4 people I got kappa value as 0.49 but when I included the 5th ...
1
vote
1answer
257 views

Using Cohen's kappa for multiple item types (e.g. binary and non-binary)

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). ...
0
votes
0answers
612 views

On the difference between Kappa and Accuracy

I have an imbalanced dataset with 5 classes. The distribution is roughly: Class 1: 0.5 Class 2: 0.25 Class 3: 0.15 Class 4: 0.05 Class 5: 0.05 I've been testing some different classification ...
0
votes
0answers
41 views

Kappa Statistic Intuition

Building Intuition | Cohens Kappa Coefficient There are several descent posts addressing this issue, but only one that gets at my main concern; When would one want to use the Kappa Statistic? This ...
1
vote
1answer
49 views

Explanation of the denominator in Cohen's kappa score?

Cohen's kappa with two raters is: $$\kappa = \frac{p_o - p_a}{1 - p_a}$$ where $p_a$ is probability of agreement by chance, and $p_o$ is the observed rate of agreement. I can't figure out why the ...
1
vote
0answers
80 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
65 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
2
votes
1answer
43 views

Relation between Kappa Index of Agreement and Cohen's kappa

I was reading a paper in which Kappa Index of Agreement (KIA) was calculated. I searched the web for a reference but I couldn't find any decent definition. My first result was instead a Wikipedia ...
0
votes
1answer
296 views

Cohen's Kappa: is it valid to average kappa for different rater pairs across multiple trials?

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of ...
0
votes
2answers
235 views

Intercoder reliability without mutually-exclusive codes

I have the results from qualitatively coding the contents of ~100 images, with 7 categories and 96 tags in total. It's incredibly difficult to define mutually exclusive codes for the content we're ...
0
votes
1answer
169 views

NaN produced from Cohen Kappa without 100% agreement

Considering the following data frame called coris_d0: ...
0
votes
1answer
759 views

Fleiss kappa in R giving strange results

I have an experiment where 4 raters gave their responses to 4 stimuli, and I need to calculate the Fleiss Kappa to check the agreements of the raters. However, I get strange results from the R ...
1
vote
0answers
888 views

How exactly is sample size determined in inter-rater reliability study?

I am conducting an inter-rater reliability study where we have four raters and the form of the rating is "Yes vs. No". Correspondingly, we chose to use Kappa value as our outcome measure. Now we want ...
2
votes
1answer
261 views

Cohen's Kappa Classifier vs. Groundtruth

The user guide on Cohen's kappa of the scikit-learn documentation states the following: The function cohen_kappa_score computes Cohen’s kappa statistic. This ...
2
votes
1answer
949 views

How to use kappa() or cohen.kappa() to check for matching assigments between observers

down vote favorite Example 1. In a common situation, if we have the following data: ...
2
votes
2answers
637 views

Cohen's kappa as a classifier “strength” estimator

What are the strengths or weaknesses of using the higher Cohen's kappa score to select a classifier from a pool of candidates? Back-story: I had a pool of candidate classifiers (~200 candidate models)...
0
votes
0answers
35 views

Adjust for rater when examining agreement or Kappa

I am conducting a 10-question survey, with responses of 'yes' and 'no' to each. I want to compare responses to a known 'gold-standard' (also with responses of yes and no). Suppose I have 100 ...
1
vote
2answers
869 views

Facing unbalanced data: AUC vs. Cohen's Kappa vs. Balanced Misclassification Rate

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest". So, I chose two ways ...
1
vote
1answer
258 views

Linear and Quadratic Weighted Kappa SPSS Error

I am using both Linear and Quadratic weighted kappa to examine the extent to which two measures (a quantitative clinical interview, and a corresponding self-report questionnaire) agree at the item ...
3
votes
1answer
3k views

Calculate Kappa statistic in R [closed]

Currently I am trying to calculate Kappa for a confusion matrix. Unfortunately, I never used Kappa in R before. I found this description how to use it: class1 A vector describing a classification of ...
0
votes
0answers
77 views

Test Retest reliability with different rater numbers

I was wondering whether its possible to run a test-retest reliability analysis, but with a different number of raters for the two time points. I have collected personality data and am looking at ...
0
votes
1answer
587 views

Compute Cohen's kappa in multi-label classification

We have 2 different interpretations for our data. Here is an example: ...
2
votes
0answers
199 views

Inter-rater reliability - when Cohen's Kappa doesn't work

I am trying to calculate inter-rater reliability for two raters using a dichotomous scale. In one part of the study, raters have to decide from a list which word candidates of a child's utterances ...
1
vote
1answer
160 views

Calculate sample size for a moderation exercise, using Cohen's Kappa

I need to conduct a moderation study using 3-4 raters, a scale of 5-10 items, and between 1 and 5 samples. Would my Kappa coefficient be meaningful with this sample size? Would I be able to generalise ...
1
vote
2answers
192 views

Von Mises mixtures: What is the maximum kappa value?

I have an R script that is estimating the concentration parameters (kappa) for Von Mises mixtures. This script is giving me values of kappa that are 60, 100 e.c.t. I was expecting kappa to be less ...
1
vote
1answer
1k views

Explain Cohen's kappa in a simplest way? [duplicate]

I'm learning statistics but don't really understand the result of kappa value posted in the first sample here. Is there anyone help me explain in English what the value 0.915 means? (observed ...
3
votes
1answer
1k views

Using Cohen's kappa statistic for evaluating a binary classifier

I am using the caret package to perform predictive modeling on a binary target variable. The outcome is very unbalanced so it is suggested to use the Kappa statistics to evaluate the binary classifier....
1
vote
0answers
31 views

Cohen's Kappa: pairs of raters out of three persons [duplicate]

For example there are 300 targets that need to be rated. There are 3 researchers (X, Y, Z). Each target should be rated by two persons. So I cut the amount of all targets into 3 blocks. Block A: ...