A message from our CEO about the future of Stack Overflow and Stack Exchange. Read now.

Questions tagged [cohens-kappa]

A measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

sample size advice for matching nominal variables

I have two databases of patients and medications. Each database has the same patients. Both have millions of records. My job is to select a sample from each and confirm that the medications match. ...
2
votes
2answers
101 views

Is there a strict relation between Accuracy and Cohen's Kappa (measures of classification quality/agreement)?

Say I have two models (classifiers), $M_1$ and $M_2$, each with their own accuracy w.r.t. the ground truth, and I calculate also Cohen's Kappa between each model and the ground truth (as a measure of ...
0
votes
0answers
7 views

Nonparametric bootstrap to construct confidence intervals for Cohen's Kappa Coefficient

I was recently asked to perform nonparametric bootstrap to construct 95% confidence intervals for $\kappa$ using normal approximation, but I'm not sure how to do this. The only data I was given was ...
2
votes
1answer
129 views

Fleiss kappa vs Cohen kappa

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood? When would one use Fleiss kappa over Cohen kappa? What are the ...
0
votes
0answers
12 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
0
votes
0answers
11 views

What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...
0
votes
0answers
17 views

Spearman's or weighted kappa?

I am hoping to do a study comparing therapist estimates of an ordinal measure (questionnaire a patient completes) with the physiotherapist's own estimates of the same thing (so the therapist will be ...
2
votes
2answers
42 views

Find if there is a correlation between two paired data points

I have two lists of data with each 10 values from 10 participants. Let’s say the first list contains IQ’s of the 10 participants. And the second list contains how many words each 10 participants can ...
0
votes
0answers
15 views

How do I calculate Inter-rater reliability between one rater and a group of raters?

So let's say I have a study in which I give a categorical rating to bunch of items. Each item could be a pair of images and I have to rate which one is brighter for example. Perhaps three ratings ...
0
votes
0answers
11 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
1
vote
1answer
21 views

weighted cohens kappa categories

I need to ask about how many categories can be included in a weighted kappa or cohens kappa statistic for inter observer reliability. Effectively we need to be reliable in assessing whether the ...
0
votes
0answers
17 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
1
vote
1answer
28 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
0
votes
0answers
10 views

Fleiss' kappa: why the proportion of agreement by chance is calculated with the observered proportions of the categories?

The formula for calculating Fleiss' kappa is: $$ \kappa = \frac{p_o - p_a}{1 - p_a} $$ The aggreement by change ratio is calculated via the sum of squares of observed category proportions: $$ p_a = ...
0
votes
0answers
13 views

Measuring Agreement between more than two coders?

I've to find inter-Coder Agreement for my data. I have around 18 coders to fill a survey for me. Now i'm confused how to find Kappa Statistic value when we have to find agreement between more than two ...
0
votes
0answers
140 views

Fleiss' Kappa: How to interpret p value > 0.05 when Kappa is < 0?

I have 3 raters who have rated 27 items on a Likert scale from 1 to 5 and want to compute Fleiss' Kappa; I do understand that a Kappa below 0 means "poor agreement" but I struggle whether this is ...
1
vote
1answer
21 views

Quadratic weighted kappa error

I have recently downloaded the extension for weighted kappa on SPSS. I followed the procedure of this video. However, at the end I have the following error "some ratings are not integers". Do you have ...
1
vote
2answers
71 views

Is it valid to report the mean of Cohen's kappa coefficients and the associated confidence intervals?

I am studying the interrater and intrarater reliability of a classification system (seven categories; nominal). I recruited 16 unique raters to classify 35 items according to the system. The raters ...
0
votes
0answers
7 views

correlation K Cohen

I have a data matrix composed by a set of stimuli (words) evaluated by two evaluators following a certain criteria. They used an ordinal scale (0, 1, 2). Is it possibile with an ordinal scale and not ...
2
votes
1answer
93 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
1answer
53 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
1
vote
1answer
226 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
0
votes
1answer
30 views

Cohen's kappa for agreements between two inter rater metrics?

I have two metrics for measuring source code cohesion. To verify that both have some agreement can I use Cohen kappa for that? Both the metrics have values in range [0 to 1]. It's not necessary if ...
3
votes
1answer
338 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
0
votes
0answers
20 views

Kappa between two groups of people

So I'm trying to compute Kappa. I've done some research on it and it seems like Cohen's Kappa is generally calculated between two people, say ...
0
votes
0answers
173 views

Difference in insight gained from Cohens' Kappa vs McNemar

I would like to understand the difference in what a test with Cohen's Kappa tells you about your data compared to what McNemar's test tells you. If Cohen's Kappa measures agreement between two ...
2
votes
0answers
35 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
1
vote
0answers
19 views

Measuring equivalence/reliability (ICC and consorts)

For a reliability study I’m administering patient reported outcomes (questionnaires) with 2 modes of administration (on paper and via an electronic device). I’d like to measure equivalence/reliability ...
0
votes
1answer
76 views

Cohen's Kappa in R versus SPSS

I was working with R and SPSS trying to determine Cohen's Kappa for a dataset. I decided to test the two and see if they would end up getting the same numbers. However, they did not. I was wondering ...
1
vote
0answers
13 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
3
votes
1answer
29 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
0answers
47 views

Cohen's Kappa and prevalence and bias in R

I am working on a dataset where two raters indicated whether or not particular features were present in the same individuals. So each individual was assessed by each of these two raters and received ...
0
votes
0answers
28 views

Is Cohen Kappa (Weighted or not) appropriate for medical research? | Agreement between measures

I'm working on a experiment when we test the effectiveness of a new diagnostic compared to an existing measure. Our results will be the format ...
1
vote
1answer
117 views

Can I compute an F1 score when the test data has no examples of one class?

I am working on a 3-class classification problem. We are cross-validating via a Leave-One-Out Approach, and there are some instances where the test data has no instances of one of my three classes. ...
0
votes
0answers
617 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
0
votes
0answers
279 views

How to deal with a negative Kappa in classification?

I have a dataset with one binary class to be predicted, with 18 binary predictors and 17400 rows. Here I used a stratified split, with approximately 85% (14648 rows) for training and 15% (2752) for ...
1
vote
1answer
61 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
2
votes
1answer
477 views

Calculation of accuracy (and Cohen's kappa) using sensitivity, specificity, positive and negative predictive values

I read How to calculate specificity from accuracy and sensitivity, but I have two diagnostic performance measures more. Please correct me if I am wrong: if Sensitivity=TP/(TP+FN) Specificity=TN/(TN+...
2
votes
2answers
143 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
1
vote
2answers
405 views

Interpreting SPSS Cohen's Kappa output

When running Cohen's Kappa in SPSS, it outputs a few things. Cohen's Kappa Asymptotic Standard Error (ASE) Approximate T Approximate Significance If I have a Kappa of 0.90 and ASE of .012, can I ...
1
vote
1answer
72 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
1
vote
1answer
419 views

Using Cohen's kappa for multiple item types (e.g. binary and non-binary)

I am looking to calculate Cohen's kappa for multiple items in a scale (15 items, with two raters). The problem is that for 12 of the items the responses available to the raters were binary (yes, no). ...
0
votes
0answers
1k views

On the difference between Kappa and Accuracy

I have an imbalanced dataset with 5 classes. The distribution is roughly: Class 1: 0.5 Class 2: 0.25 Class 3: 0.15 Class 4: 0.05 Class 5: 0.05 I've been testing some different classification ...
1
vote
1answer
71 views

Explanation of the denominator in Cohen's kappa score?

Cohen's kappa with two raters is: $$\kappa = \frac{p_o - p_a}{1 - p_a}$$ where $p_a$ is probability of agreement by chance, and $p_o$ is the observed rate of agreement. I can't figure out why the ...
1
vote
0answers
136 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
74 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
2
votes
1answer
63 views

Relation between Kappa Index of Agreement and Cohen's kappa

I was reading a paper in which Kappa Index of Agreement (KIA) was calculated. I searched the web for a reference but I couldn't find any decent definition. My first result was instead a Wikipedia ...
1
vote
1answer
495 views

Cohen's Kappa: is it valid to average kappa for different rater pairs across multiple trials?

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of ...
0
votes
2answers
545 views

Intercoder reliability without mutually-exclusive codes

I have the results from qualitatively coding the contents of ~100 images, with 7 categories and 96 tags in total. It's incredibly difficult to define mutually exclusive codes for the content we're ...
0
votes
1answer
291 views

NaN produced from Cohen Kappa without 100% agreement

Considering the following data frame called coris_d0: ...