Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

learn more… | top users | synonyms

2
votes
1answer
395 views

Sample size needed for Fleiss' Kappa?

A group of raters (about 20) will be watching a series of videos and will be classifying them into 4 categories. I will be running a Fleiss' kappa to measure the agreement. How does one compute for ...
4
votes
0answers
379 views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
3
votes
0answers
30 views

Is there such a thing as a multidimensional kappa test?

I am involved with designing a new diagnosis method for a routine clinical procedure. I want to compare the agreement between operators to the agreement between methods. Is there such a thing as a ...
2
votes
0answers
81 views

Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
2
votes
0answers
145 views

Inter-rater agreement with multiple raters

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
2
votes
0answers
209 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
2
votes
0answers
138 views

Validity of grand mean intra-rater Kappa or ICC

For 5 raters, I have calculated intra-rater Cohen's Kappa statistics for the test-retest nominal ratings, intra-rater Kendall's tau-b statistics for the test-retest ordinal ratings, and intra-rater ...
2
votes
0answers
297 views

How should I interpret the Kappa Fleiss test?

I noticed that when I have tables which the values are only 0 and 1, I get a kappa of 1 when the table is completely full of one, and when I have a table of zeros I get NaN as result using the irr ...
2
votes
0answers
112 views

Is it statistically correct and safe to calculate Kappa 2 test means of combinations of 2 against the value of a Fleiss' Kappa?

For my problem I have a group of p participants and s subjects. I was suggested to calculate the Kappa Cohen Test on ...
2
votes
0answers
78 views

What is currently the more accepted Agreement Between Raters Statistics or most referenced paper?

I found a question that is related here, but it doesn't really goes on what I want to know. I found a couple of papers using Kappa Statistic from 2006, and 2010, but afterwards I found other authors ...
1
vote
0answers
18 views

Combining different raters scores

In my experiment two raters evaluate 100 products, both expressing an "innovation score" on a likert scale (range is 1 to 5, where 1 is "not innovative" and 5 is "disruptive innovation"). I ...
1
vote
0answers
7 views

Inter-rater reliability for $k$ raters with $0$ to $n$ of $n$ categories applying to each subject

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
1
vote
0answers
47 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
1
vote
0answers
56 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
1
vote
0answers
32 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
1
vote
0answers
37 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
1
vote
0answers
74 views

Is Fleiss's $\kappa$ as conservative as Cohen's $\kappa$?

I know Cohen's $\kappa$ for inter-rater reliability is often regarded as a conservative estimation--e.g., Strijbos & al. below, who say The higher the number of categories, the smaller the ...
1
vote
0answers
83 views

When should I prefer kappa over accuracy for binary classification problems?

I'm using the caret package in R that provides two kind of metrics for classification problems - accuracy and kappa. When should ...
0
votes
0answers
11 views

Why does Fleiss's $\kappa$ decrease with increased response homogeneity?

The Fleiss method for computing inter-rater agreement seems like a great measure for computing agreement among multiple raters, in terms of agreement by chance. However, I've noticed that when ...
0
votes
0answers
16 views

Weighted kappa: kappa2 (irr) vs cohen.kappa (psych)

I want to calculate the weighted Kappa value with R. I compared two functions kappa2 and cohen.kappa but the two kappa values disagree. Is there an explanation for this or am I doing sth wrong? My ...
0
votes
0answers
13 views

Model Performance - Accuracy not significant but Kappa is?

I am comparing an SVM model to a logistic regression model. When I run a comparison between these models, I get the following output: Accuracy: SVM vs Logistic: p=0.7882 Kappa : SVM vs Logistic: ...
0
votes
0answers
7 views

How can this rating has negative Fleiss Kappa?

I am trying to calculate the Fleiss' Kappa of the following data and I got the result: ...
0
votes
0answers
5 views

Inner annotation agreement

I am currently working on an NLP project where I had to mark an initial list of words, giving a score of either 1 or 2 to each word in the list. I got the list marked from 2 people and found that the ...
0
votes
0answers
25 views

Weighted kappa with quadratic weighting and ICC not giving the same answer

I have 10 subjects who filled questionnaire of 28 items. Each item has 5 options (scale 1 to 5). However when I compute in SPSS using weighted kappa with quadratic weighing versus ICC two-way mixed ...
0
votes
0answers
7 views

Comparing TrueScore with the System Score

I have collected three human raters scores for an essay question(N=54). I am building Automated Scoring Algorithem. So I need to compare my machine score with the Human rater score. All are using 0-10 ...
0
votes
0answers
73 views

Comparison test of reapeated Cohen's kappa

Suppose two raters judged a set of stimuli. Then Cohen's kappa was calculated as measure of inter-rater agreement ($\kappa_{pre-training}$). Suppose now that after a training session, the same two ...
0
votes
0answers
287 views

Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where ...
0
votes
0answers
252 views

Software to compute p-values and Cohen's kappa for 10 by 10 contingency table?

I have a 10 by 10 contingency table with most of my cells = 0, and my total cell count = 70. Is there a statistical package or online calculator which I can use to obtain p-values and Cohen's kappa?