Questions tagged [agreement-statistics]
Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.
435
questions
1
vote
0
answers
29
views
Negative ICC values for inter-rater reliability
I have conducted research which assessed the technical skills of surgeons.
A surgeon will perform a simulated operation, and be assessed by two different assessors whilst completing the procedure. The ...
- 11
1
vote
0
answers
17
views
Confused on Cohen's Kappa - Exploring Rater Agreement Stats
I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
- 11
1
vote
0
answers
39
views
Inter-rater reliability of two raters for conversation analysis (multiple categories with non-binary values)
Two raters have done conversation analysis, coding all the topics in the interaction for multiple participants. There are multiple topics, but they are predetermined. There are two raters, and both ...
- 87
0
votes
0
answers
11
views
Calculating Inter-Observer Reliability when not all categories are scored
I have two main questions:
1) Is there a way to calculate IRR when the range of scores differ between observers? If not, how do you get IRR without excluding data?
I am calculating inter-observer ...
- 13
0
votes
0
answers
24
views
Preferred statistics to evaluate interrater agreement of group of raters (nominal dependent variable, two groups of raters) and compare both groups
I hypothesized that a set of symptoms is not really specific to a particular syndrome (let´s say A), so I recruited two groups of experts to classify each symptom (I have 16) to condition A, B, C (...
1
vote
1
answer
29
views
How can I compare two methods of measuring variables for a set of objects?
I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
1
vote
1
answer
26
views
Fleiss Kappa score - NaN for perfect agreement
I am working on a dataset that has three raters. Rating are Yes/No.
I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return ...
- 111
3
votes
0
answers
30
views
Difference between kappa statistics
I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
- 31
0
votes
0
answers
8
views
Inter-rater reliability against multiple raters
I am conducting inter-rater reliability by doing a clinical test of 25 participants. Can I do the reliability against multiple raters? I mean that I will do the test for all the participants and then ...
0
votes
0
answers
15
views
Can you use inter-rater reliability to measure reliability of a whole measure with items measuring different things? (Krippendorff's Alpha or similar)
Let's say that you have one case study, where 30 raters are asked to apply a measure that has 5 items. Can you compute a single Krippendorff's Alpha (kalpha) value across all 5 items (that are the ...
- 1
0
votes
0
answers
29
views
Calculate minimum sample size for Fleiss Kappa
Can you please explain to me how to calculate a minimum sample size (rated subjects) for Fleiss Kappa? I cannot find neither calculators to do this, nor some reference values.
Thank you very much.
G
- 1
5
votes
1
answer
67
views
What's the standard deviation of the difference between randomly paired data?
Assume that we want to compare the agreement of two methods of measuring blood pressure (e.g. a machine and a sphygmomanometer). In their paper, Preiss & Fisher (2008) discuss the standard ...
- 29k
1
vote
1
answer
59
views
Any good metric for measuring multi-annotator agreement on an imbalanced dataset?
Is there an agreement method that would be well-suited for a data annotation task where:
the labels are discrete classes
each datapoint belongs to exactly one class (multi-class classification)
each ...
- 33
0
votes
0
answers
15
views
Calculate Fleiss Kappa with annotator replacement
If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
- 1
0
votes
0
answers
19
views
Can you use Krippendorff's Alpha to assess reliability of an entire measure using one case study?
I am interested in using Krippendorff's Alpha for a dataset containing a large number of raters to assess the reliability of using an assessment tool for case studies, which contains 8 items with ...
- 1
0
votes
0
answers
18
views
Asymmetric confidence intervals for Cohen's kappa
I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
- 277
0
votes
0
answers
28
views
Is there such thing as inter-annotator agreement for a set of rankings?
I'm familiar with inter-annotator agreement (or inter-rater reliability) metrics for data that has categorical annotations, but what about a set of data samples that are ranked by several annotators? ...
- 3,346
0
votes
0
answers
64
views
What is the correct Kappa coefficient to use when more than two labelers are present?
I have a dataset consisting of m rows and 2 columns. The m rows denote a patient and the columns denote the label assigned to them. The label size can be arbitrarily long, with labelers given the ...
- 500
0
votes
0
answers
15
views
Agreement statistic for likert scale evaluation of different image modalities
I have some data of dental images. The same image was taken using the gold standard method and using an alternative new more cost effective method. Evaluators rate each image (blinded) based on a ...
- 21
0
votes
0
answers
69
views
Calculating Cohen's Kappa for a dataset when categories differ between examples
Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
0
votes
0
answers
65
views
How many items for a reliable Fleiss kappa?
Suppose that I have N items that were rated by a first group of annotators on a scale from 1 to 5 (each item was rated only once). I want to compute an inter-annotator agreement score using Fleiss ...
1
vote
1
answer
52
views
Chance Agreement for Fleiss Kappa
In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as
where $p_j$ is defined as the proportion of ratings which were to ...
- 31
1
vote
0
answers
31
views
Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]
I an assessing the agreement between two measurements methods on a binary presence/absence metric.
Here is one example of a contingency table:
...
- 249
0
votes
0
answers
44
views
An overall agreement coefficient for each subject
There are two paired data sets. Each suject has 80 variables gathered from CT scan and the same 80 variables gathered from MRI. How to obtain a global agreement coefficient (Inter-rater reliability) ...
0
votes
0
answers
38
views
Interrater agreement for rankings on multiple objects
I asked 5 raters to rate 3 differently styled images from a total of 100 people (styles such as b&w, highly saturated, and natural colors).
Since each person has 3 images attached, each rater ...
- 60
0
votes
0
answers
18
views
Measuring inter-rater agreement for unclear number of events
I have 2 raters who watch videos of test sessions and annotate emotions. There is a defined number of categories to choose from. But there is no defined number of situations they have to annotate. So ...
- 1
0
votes
0
answers
12
views
Text Labeling with 5 Annotators
for a study I let participants classify a text into 3 categories [0,1,2]. Every text got 5 annotations.
So my assumption is now, that the score of Krippendorff's Alpha or Fleiss Kappa decreases, the ...
0
votes
0
answers
51
views
Intra-rater reliability of an instrument
I'm working on the implementation of a "new version" of an existing instrument in the posture analysis.
I know that if I want to compare the measures between the old one and the new one I ...
- 23
1
vote
0
answers
60
views
Inter annotator agreement with varying annotators [duplicate]
I run a human evaluation on a recruiting website.
The situation is the following:
There are 78 annotators.
There are 100 questions.
Each annotator annotates 5 questions.
So, on average*, each ...
- 83
0
votes
0
answers
12
views
comparing images and narratives for generating an emotion
I am a psychology student. I am doing my dissertation on the ability of images and narratives to generate emotion. I am planning to compare the two. Before doing the same, I am getting the two rated ...
- 1
0
votes
0
answers
20
views
What intraclass correlation setup to use when there are a variable $k$ number of raters?
I have 200 subjects that need rating, each of which are given to between 3 to 8 raters (a rater may be able to grade more than one subject). I would like to compute the Intraclass Correlation but do ...
- 3,422
0
votes
0
answers
42
views
How to interpret Fleiss' kappa?
How do you interpret Fleiss' kappa? Can you compare Fleiss' kappa values when the amount of raters differ? For example, Fleiss' kappa with three raters versus a value obtained with more/less raters.
0
votes
0
answers
82
views
How to do sample size calculation for Krippendorff's alpha (paired nominal/categorized data with 10 raters/observers)?
I'm planning an prospective observational trial studying the agreement of 10 doctors (from the same field) in diagnosing certain findings on ultrasound (nominal yes/no outcomes). There will inevitably ...
- 11
1
vote
1
answer
87
views
Pair confusion matrix for more than two cluster algorithms
How would one code the following issue:
The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
0
votes
1
answer
58
views
Analyzing method comparison studies that include repeated measurements
I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison ...
- 1
0
votes
0
answers
15
views
Why is my concordance measure greater than 1?
I've written a function to assess the coefficient of concordance for vague data. I'm using it to measure consistency in the rank order of some items ranked by two reviewers. It's vague because each ...
- 901
0
votes
1
answer
41
views
Should we test only 20% of the sample for intraclass correlation coefficient?
I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post ...
- 641
2
votes
1
answer
127
views
inter-rater reliability for numeric variables
I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
- 641
1
vote
1
answer
32
views
Inter-rater reliability (complex survey )
Help calculating Inter-rater reliability in a complex online survey (free text).
The questions are free text. We have now coded the potential answers. Examples:
1.) Hobbies (Time in nature, mind-body ...
- 11
0
votes
0
answers
10
views
Kappa with one judgment
Is it possibile to use the cohen's Kappa or the Fleiss' Kappa when you have only one judgment for every rater?
I don't think so because I don't have a contingency table.
Is there a different way to ...
- 143
0
votes
0
answers
47
views
Fleiss' Kappa raters limit
I have 20 raters for 5 items and I need to calculate the inter-rater agreement for every item.
I obtain the same results with unweighted, linear and quadratic weights in the items with 3 categories (a ...
- 143
1
vote
1
answer
249
views
Reporting results of random seeds and bootstrapping (stats)
I am trying to compare my model's (a neural network) results with human-scored results on the same type of data. However, there are no ground truth labels (humans disagree on classifying these results)...
- 25
5
votes
1
answer
847
views
Measures to compare clustering partitions
What are the most used measures (coefficients) to compare two partitions of objects into clusters? I am speaking of validating the results of clustering, not of classifying; the measures known as ...
- 56k
4
votes
1
answer
349
views
Measures to compare classification partitions
What are the most used measures (coefficients) to compare two partitions of objects into classes? I am speaking of validating the results of classification, not of clustering; the measures known as ...
- 56k
1
vote
1
answer
149
views
What is the right statistical measure for consistency?
I have ordinal data from a Likert survey with $45$ questions and $15$ participants. Every participant took the survey six times, so I have six answers from every participant for every question. All ...
- 143
1
vote
1
answer
64
views
How to measure the "correlation" between two sequences of objects ordered by different criteria, considering higher positions as higher weights?
I suppose this is a pretty common problem since when we use ranks we are often more interested in how similar they are at the top (or at the bottom, or both extremes) than in how they compare in the ...
- 255
0
votes
0
answers
38
views
how to calculate an agreement value in a vote between multiple people?
I am not at all good with statistics which also doesn't help me looking for a proper answer.
I am trying to calculate an agreement value in a vote between multiple people. And people can vote in ...
- 1
3
votes
1
answer
58
views
assessing which random sample agrees more with a preferred ranking
I am not a mathematician or a statistician. But, I think the question I have is related to statistics. I will start with a made up example.
If I can grade apples into,say four grades from 1 to 4, one ...
- 33
0
votes
0
answers
38
views
Can there be a single limit of agreement value in Bland-Altman analysis?
While trying to learn Bland-Altman analysis, I came across a situation that confused me a bit. I can understand upper (mean + 2 SD) and lower limits (mean – 2 SD), but a single limit of agreement (LOA)...
- 71
1
vote
1
answer
147
views
How to determine agreement between clustering methods?
Let's say you want to compare the outcome of KMeans and KMedoids. How to determine if cluster 1 from KMeans can be compared with cluster 1 with KMedoids. Or, in other words, let's say KMeans labels ...