# Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

449 questions
Filter by
Sorted by
Tagged with
9 views

### Per-Item inter rater reliability with multiple values and raters

I am trying to find the best statistical calculation to measure the agreement between 72 different raters on one item. My goal is to convey in a statistic how spread the raters are on their rating and ...
• 101
30 views

### Is it possible to use unweighted Kappa for some questions and weighted for others to measure interrater agreement in the same questionnaire?

I intend to conduct an interrater agreement analysis on a questionnaire about student essays that contains binary options, mainly yes-no questions, ordinal variables, primarily a likert agreement ...
14 views

### What is the best metric to use to discard annotators with low IAA (inter-annotator agreement) with all others?

This question is specific to ordinal data collected on the likert scale What is the best metric to discard annotators with low inter-annotator agreement (IAA) with others? from e.g., Cohen’s Kappa, ...
• 141
12 views

### How is Krippendorff's alpha defined when expected disagreement is 0?

I'm wondering what is the definition of Krippendorff's Alpha statistics when the expected disagreement is 0? The general form of Krippendorff's Alpha is: $\alpha = 1 - \frac{D_{o}}{D_{e}}$ I'm ...
1 vote
63 views

### Calculate inter-rater noise using Kahnemans (2021) approach

I need help calculating signal and noise based on the method described by Kahneman et al. (2021) in their book "Noise." They provide a technique for quantifying noise between raters ...
67 views

### How to improve testing method B, if Bland-Altman analysis show a near-perfect correlation between the difference and the mean between method A and B?

I have two sets of measurements (two different methods), A and B. Correlation between the measurements is only modest. Therefore, I wondered whether there is some inherent bias in (one or both) of ...
1 vote
20 views

### How to calculate an ICC for test-retest, with more than one rater

In my data, i have two raters that rated multiple scores to do a test-retest analysis. The method fulfill the requirements for test-retest (short period between both visits) I would like to calculate ...
• 143
13 views

### Is it possible to calculate inter-rater reliability for one item rated by multiple raters with weights?

I have a survey with a number of statements that had participants categorizing the statements into one of the 4 options they were provided. The participants were then asked to rate the confidence of ...
83 views

### Comparing human estimates and algorithm on normalized score

By means of an algorithm, we want to predict a metric value $Y$ that can vary for different items $i=1,\ldots,n$. The quantities $y_i$ have been estimated by $N$ human annotators and we want to ...
• 5,392
63 views

### method comparison/agreement - is Bland-Altman or equivalence of the mean best

I am interested in the appropriate technique for assessing the agreement of the paired values of measurements made by 2 measuring devices, an equivalence test of the mean of the difference in the ...
• 360
23 views

### How should I approach calculating sample size for number of raters required for a study?

I am proposing a study for evaluating reports generated by 3 human experts vs 3 different LLMs on a set of 10 situations. Basically, we're trying to whether the human experts are better or if the LLMs ...
20 views

### How to format data for one-way intraclass correlation coefficient?

I'm testing inter-rater reliability of an instrument on a sample of subjects from several sites. Each subject is assessed by two raters, different between each site. According to this paper, I should ...
• 142
40 views

### Correct approach for evaluating correlation between multiple measurement devices used on the same subject (repeated measures)

My scenario is as follows. I am using three different devices (TempDeviceA, TempDeviceB, and TempDeviceC) to take the temperature of each animal in a group every day over several weeks. TempDeviceC is ...
49 views

### Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
• 151
1 vote
28 views

### Extending paired t-test to compare agreement between three or more analytical instruments over time

I have three instruments that measure the concentration of dust in the air, and I want to test whether the mean difference between their measurements is zero (i.e that they give the same measurement). ...
14 views

### How to determine goodness-of-fit between non-parametric 2d-datasets

Lets say I have a set of paired x' and y' values and I have a N sets of reference values also consisting of paired x and y values. I would like to determine which reference set best matches by x'y' ...
1 vote
17 views

### Are agreement statistics suitable for comparing ratings produced by the same rater on different experiments?

I am interested in comparing the results of a single statistical test applied separately to two different datasets. My test produces lists of p-values and estimated effect sizes for the features of a ...
• 143
31 views

### What is a good metric for agreement in this problem?

Suppose I have a dataset that needs annotations. I have about $5000$ total $x$ values, and annotations by humans are needed to create $y$ values. $y$ takes a value in $\{0, 1\}$. For example, consider ...
• 101
1 vote
36 views

### Estimate sensitivity and specificity from inter-rater agreement?

Assuming two binary (Y in {0, 1}) annotators or classifiers (A and B), that are: Conditionally independent, i.e. P(A=0, B=0|Y=1) = P(A=0|Y=1)*P(B=0|Y=1) and the same for Y=0. Better than random, i.e. ...
• 11
1 vote
26 views

### Evaluate two datasets for corresponding to given pairwise matching

Suppose I have two matched $N \times D$ datasets $X$ and $Y$. The samples are in matched order (with $n$th sample in $X$ being the same object as the $n$th sample in $Y$). The features are (or at ...
1 vote
14 views

### Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
• 11
92 views

### Inter-rater reliability for mixed data types?

I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my ...
• 63
144 views

### Pearson/Spearman correlation or intraclass correlation coefficient?

I wish to study the correlation of responses of 150 participants to two concurrently-administered previously-validated questionnaires (each giving numerical summary scores as output). Both ...
1 vote
119 views

### How to compare Gwet's ac1 coefficient between group of raters?

So I have 4 raters who scored the subjects in the study with a nominal score like: "Present", "maybe", and "absent". I made pairs of the raters and evaluated their inter-...
528 views

### How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
• 49
52 views

### How to measure consistency of teacher grading?

I would like to determine how consistent teacher grades are for a given item, both for that teacher and across all teachers. E.g., does teacher A consistently score item 1 as a 1,2,3, or 4; and do all ...
25 views

### McNemar test or similar with multiple pairings?

Let's say I have test condition A and B, and a binary classification model. I have paired samples in conditions A and B, and I want to test if these conditions are different in classifier performance. ...
1 vote
92 views

### Testing if refusal rates are "correlated" between 3 judges

I have data looking like this: group refusal proportion (judge A) refusal proportion (judge B) refusal proportion (judge C) group 1 0.2 0.3 0.05 group 2 0.1 0.5 0.6 group 3 0.6 0.9 0.7 group 4 0....
• 57
46 views

### Accounting for occasionally correlated data in performance analysis

I'm doing a study in which a new diagnostic method is evaluated. Samples are taken from patients, both this new method and the current standard method are applied, and we calculate concordance between ...
• 101
41 views

### Estimating error rates from inter-rater reliability values

I have been tasked to approximate the error rate of a research procedure. The only data I have available are reports which specify the number of targets (n), the number of raters (k) and give pairwise ...
1 vote
52 views

### Negative ICC values for inter-rater reliability

I have conducted research which assessed the technical skills of surgeons. A surgeon will perform a simulated operation, and be assessed by two different assessors whilst completing the procedure. The ...
• 11
1 vote
45 views

### Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
1 vote
47 views

### Inter-rater reliability of two raters for conversation analysis (multiple categories with non-binary values)

Two raters have done conversation analysis, coding all the topics in the interaction for multiple participants. There are multiple topics, but they are predetermined. There are two raters, and both ...
• 97
27 views

### Calculating Inter-Observer Reliability when not all categories are scored

I have two main questions: 1) Is there a way to calculate IRR when the range of scores differ between observers? If not, how do you get IRR without excluding data? I am calculating inter-observer ...
• 13
33 views

### Preferred statistics to evaluate interrater agreement of group of raters (nominal dependent variable, two groups of raters) and compare both groups

I hypothesized that a set of symptoms is not really specific to a particular syndrome (let´s say A), so I recruited two groups of experts to classify each symptom (I have 16) to condition A, B, C (...
1 vote
34 views

### How can I compare two methods of measuring variables for a set of objects?

I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
1 vote
91 views

### Fleiss Kappa score - NaN for perfect agreement

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return ...
• 111
51 views

### Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
• 31
91 views

### What's the standard deviation of the difference between randomly paired data?

Assume that we want to compare the agreement of two methods of measuring blood pressure (e.g. a machine and a sphygmomanometer). In their paper, Preiss & Fisher (2008) discuss the standard ...
• 31k
296 views

### Any good metric for measuring multi-annotator agreement on an imbalanced dataset?

Is there an agreement method that would be well-suited for a data annotation task where: the labels are discrete classes each datapoint belongs to exactly one class (multi-class classification) each ...
• 143
30 views

### Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
52 views

### Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
132 views

### Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
93 views

### How many items for a reliable Fleiss kappa?

Suppose that I have N items that were rated by a first group of annotators on a scale from 1 to 5 (each item was rated only once). I want to compute an inter-annotator agreement score using Fleiss ...
1 vote
105 views

### Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
1 vote
51 views

### Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
• 309
77 views

### Intra-rater reliability of an instrument

I'm working on the implementation of a "new version" of an existing instrument in the posture analysis. I know that if I want to compare the measures between the old one and the new one I ...
• 23
1 vote
63 views

### Inter annotator agreement with varying annotators [duplicate]

I run a human evaluation on a recruiting website. The situation is the following: There are 78 annotators. There are 100 questions. Each annotator annotates 5 questions. So, on average*, each ...
• 83
1 vote
110 views

### Pair confusion matrix for more than two cluster algorithms

How would one code the following issue: The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
101 views

### Analyzing method comparison studies that include repeated measurements

I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison ...
• 1