# Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

443 questions
Filter by
Sorted by
Tagged with
1 vote
20 views

### Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
• 111
1 vote
25 views

### Extending paired t-test to compare agreement between three or more analytical instruments over time

I have three instruments that measure the concentration of dust in the air, and I want to test whether the mean difference between their measurements is zero (i.e that they give the same measurement). ...
11 views

### How to determine goodness-of-fit between non-parametric 2d-datasets

Lets say I have a set of paired x' and y' values and I have a N sets of reference values also consisting of paired x and y values. I would like to determine which reference set best matches by x'y' ...
1 vote
14 views

### Are agreement statistics suitable for comparing ratings produced by the same rater on different experiments?

I am interested in comparing the results of a single statistical test applied separately to two different datasets. My test produces lists of p-values and estimated effect sizes for the features of a ...
• 143
29 views

### What is a good metric for agreement in this problem?

Suppose I have a dataset that needs annotations. I have about $5000$ total $x$ values, and annotations by humans are needed to create $y$ values. $y$ takes a value in $\{0, 1\}$. For example, consider ...
• 101
1 vote
31 views

### Estimate sensitivity and specificity from inter-rater agreement?

Assuming two binary (Y in {0, 1}) annotators or classifiers (A and B), that are: Conditionally independent, i.e. P(A=0, B=0|Y=1) = P(A=0|Y=1)*P(B=0|Y=1) and the same for Y=0. Better than random, i.e. ...
• 11
1 vote
24 views

### Evaluate two datasets for corresponding to given pairwise matching

Suppose I have two matched $N \times D$ datasets $X$ and $Y$. The samples are in matched order (with $n$th sample in $X$ being the same object as the $n$th sample in $Y$). The features are (or at ...
1 vote
12 views

### Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
• 11
87 views

### Inter-rater reliability for mixed data types?

I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my ...
• 63
101 views

### Pearson/Spearman correlation or intraclass correlation coefficient?

I wish to study the correlation of responses of 150 participants to two concurrently-administered previously-validated questionnaires (each giving numerical summary scores as output). Both ...
1 vote
66 views

### How to compare Gwet's ac1 coefficient between group of raters?

So I have 4 raters who scored the subjects in the study with a nominal score like: "Present", "maybe", and "absent". I made pairs of the raters and evaluated their inter-...
400 views

### How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
• 49
46 views

### How to measure consistency of teacher grading?

I would like to determine how consistent teacher grades are for a given item, both for that teacher and across all teachers. E.g., does teacher A consistently score item 1 as a 1,2,3, or 4; and do all ...
13 views

### McNemar test or similar with multiple pairings?

Let's say I have test condition A and B, and a binary classification model. I have paired samples in conditions A and B, and I want to test if these conditions are different in classifier performance. ...
1 vote
91 views

### Testing if refusal rates are "correlated" between 3 judges

I have data looking like this: group refusal proportion (judge A) refusal proportion (judge B) refusal proportion (judge C) group 1 0.2 0.3 0.05 group 2 0.1 0.5 0.6 group 3 0.6 0.9 0.7 group 4 0....
• 57
46 views

### Accounting for occasionally correlated data in performance analysis

I'm doing a study in which a new diagnostic method is evaluated. Samples are taken from patients, both this new method and the current standard method are applied, and we calculate concordance between ...
• 101
39 views

### Estimating error rates from inter-rater reliability values

I have been tasked to approximate the error rate of a research procedure. The only data I have available are reports which specify the number of targets (n), the number of raters (k) and give pairwise ...
1 vote
42 views

### Negative ICC values for inter-rater reliability

I have conducted research which assessed the technical skills of surgeons. A surgeon will perform a simulated operation, and be assessed by two different assessors whilst completing the procedure. The ...
• 11
1 vote
33 views

### Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
1 vote
42 views

### Inter-rater reliability of two raters for conversation analysis (multiple categories with non-binary values)

Two raters have done conversation analysis, coding all the topics in the interaction for multiple participants. There are multiple topics, but they are predetermined. There are two raters, and both ...
• 97
19 views

### Calculating Inter-Observer Reliability when not all categories are scored

I have two main questions: 1) Is there a way to calculate IRR when the range of scores differ between observers? If not, how do you get IRR without excluding data? I am calculating inter-observer ...
• 13
30 views

### Preferred statistics to evaluate interrater agreement of group of raters (nominal dependent variable, two groups of raters) and compare both groups

I hypothesized that a set of symptoms is not really specific to a particular syndrome (let´s say A), so I recruited two groups of experts to classify each symptom (I have 16) to condition A, B, C (...
1 vote
34 views

### How can I compare two methods of measuring variables for a set of objects?

I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
1 vote
65 views

### Fleiss Kappa score - NaN for perfect agreement

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return ...
• 111
39 views

### Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
• 31
9 views

### Inter-rater reliability against multiple raters

I am conducting inter-rater reliability by doing a clinical test of 25 participants. Can I do the reliability against multiple raters? I mean that I will do the test for all the participants and then ...
16 views

### Can you use inter-rater reliability to measure reliability of a whole measure with items measuring different things? (Krippendorff's Alpha or similar)

Let's say that you have one case study, where 30 raters are asked to apply a measure that has 5 items. Can you compute a single Krippendorff's Alpha (kalpha) value across all 5 items (that are the ...
86 views

### Calculate minimum sample size for Fleiss Kappa

Can you please explain to me how to calculate a minimum sample size (rated subjects) for Fleiss Kappa? I cannot find neither calculators to do this, nor some reference values. Thank you very much. G
77 views

### What's the standard deviation of the difference between randomly paired data?

Assume that we want to compare the agreement of two methods of measuring blood pressure (e.g. a machine and a sphygmomanometer). In their paper, Preiss & Fisher (2008) discuss the standard ...
• 30.1k
186 views

### Any good metric for measuring multi-annotator agreement on an imbalanced dataset?

Is there an agreement method that would be well-suited for a data annotation task where: the labels are discrete classes each datapoint belongs to exactly one class (multi-class classification) each ...
• 143
22 views

### Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
23 views

### Can you use Krippendorff's Alpha to assess reliability of an entire measure using one case study?

I am interested in using Krippendorff's Alpha for a dataset containing a large number of raters to assess the reliability of using an assessment tool for case studies, which contains 8 items with ...
43 views

### Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
38 views

### Is there such thing as inter-annotator agreement for a set of rankings?

I'm familiar with inter-annotator agreement (or inter-rater reliability) metrics for data that has categorical annotations, but what about a set of data samples that are ranked by several annotators? ...
• 3,787
82 views

### What is the correct Kappa coefficient to use when more than two labelers are present?

I have a dataset consisting of m rows and 2 columns. The m rows denote a patient and the columns denote the label assigned to them. The label size can be arbitrarily long, with labelers given the ...
16 views

### Agreement statistic for likert scale evaluation of different image modalities

I have some data of dental images. The same image was taken using the gold standard method and using an alternative new more cost effective method. Evaluators rate each image (blinded) based on a ...
120 views

### Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
85 views

### How many items for a reliable Fleiss kappa?

Suppose that I have N items that were rated by a first group of annotators on a scale from 1 to 5 (each item was rated only once). I want to compute an inter-annotator agreement score using Fleiss ...
1 vote
84 views

### Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
1 vote
42 views

### Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
• 309
73 views

### Intra-rater reliability of an instrument

I'm working on the implementation of a "new version" of an existing instrument in the posture analysis. I know that if I want to compare the measures between the old one and the new one I ...
• 23
1 vote
61 views

### Inter annotator agreement with varying annotators [duplicate]

I run a human evaluation on a recruiting website. The situation is the following: There are 78 annotators. There are 100 questions. Each annotator annotates 5 questions. So, on average*, each ...
• 83
1 vote
98 views

### Pair confusion matrix for more than two cluster algorithms

How would one code the following issue: The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
70 views

### Analyzing method comparison studies that include repeated measurements

I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison ...
• 1
1 vote
83 views

### Should we test only 20% of the sample for intraclass correlation coefficient?

I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post ...
• 721
291 views

### inter-rater reliability for numeric variables

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
• 721
1 vote
37 views

### Inter-rater reliability (complex survey )

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body ...
• 11
1 vote
315 views

### Reporting results of random seeds and bootstrapping (stats)

I am trying to compare my model's (a neural network) results with human-scored results on the same type of data. However, there are no ground truth labels (humans disagree on classifying these results)...
1k views

### Measures to compare clustering partitions

What are the most used measures (coefficients) to compare two partitions of objects into clusters? I am speaking of validating the results of clustering, not of classifying; the measures known as ...
• 57.2k