Skip to main content

Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Per-Item inter rater reliability with multiple values and raters

I am trying to find the best statistical calculation to measure the agreement between 72 different raters on one item. My goal is to convey in a statistic how spread the raters are on their rating and ...
Jaromando's user avatar
  • 101
0 votes
0 answers
30 views

Is it possible to use unweighted Kappa for some questions and weighted for others to measure interrater agreement in the same questionnaire?

I intend to conduct an interrater agreement analysis on a questionnaire about student essays that contains binary options, mainly yes-no questions, ordinal variables, primarily a likert agreement ...
Christopher Michael Turner Mue's user avatar
2 votes
0 answers
14 views

What is the best metric to use to discard annotators with low IAA (inter-annotator agreement) with all others?

This question is specific to ordinal data collected on the likert scale What is the best metric to discard annotators with low inter-annotator agreement (IAA) with others? from e.g., Cohen’s Kappa, ...
user2160809's user avatar
0 votes
0 answers
12 views

How is Krippendorff's alpha defined when expected disagreement is 0?

I'm wondering what is the definition of Krippendorff's Alpha statistics when the expected disagreement is 0? The general form of Krippendorff's Alpha is: $\alpha = 1 - \frac{D_{o}}{D_{e}}$ I'm ...
Vera Bernhard's user avatar
1 vote
0 answers
63 views

Calculate inter-rater noise using Kahnemans (2021) approach

I need help calculating signal and noise based on the method described by Kahneman et al. (2021) in their book "Noise." They provide a technique for quantifying noise between raters ...
Magnus Nordmo's user avatar
3 votes
1 answer
67 views

How to improve testing method B, if Bland-Altman analysis show a near-perfect correlation between the difference and the mean between method A and B?

I have two sets of measurements (two different methods), A and B. Correlation between the measurements is only modest. Therefore, I wondered whether there is some inherent bias in (one or both) of ...
pishcotec_a's user avatar
1 vote
0 answers
20 views

How to calculate an ICC for test-retest, with more than one rater

In my data, i have two raters that rated multiple scores to do a test-retest analysis. The method fulfill the requirements for test-retest (short period between both visits) I would like to calculate ...
BPeif's user avatar
  • 143
0 votes
0 answers
13 views

Is it possible to calculate inter-rater reliability for one item rated by multiple raters with weights?

I have a survey with a number of statements that had participants categorizing the statements into one of the 4 options they were provided. The participants were then asked to rate the confidence of ...
Jay Jakka's user avatar
2 votes
0 answers
83 views

Comparing human estimates and algorithm on normalized score

By means of an algorithm, we want to predict a metric value $Y$ that can vary for different items $i=1,\ldots,n$. The quantities $y_i$ have been estimated by $N$ human annotators and we want to ...
cdalitz's user avatar
  • 5,392
2 votes
1 answer
63 views

method comparison/agreement - is Bland-Altman or equivalence of the mean best

I am interested in the appropriate technique for assessing the agreement of the paired values of measurements made by 2 measuring devices, an equivalence test of the mean of the difference in the ...
user3156942's user avatar
0 votes
0 answers
23 views

How should I approach calculating sample size for number of raters required for a study?

I am proposing a study for evaluating reports generated by 3 human experts vs 3 different LLMs on a set of 10 situations. Basically, we're trying to whether the human experts are better or if the LLMs ...
user2615936's user avatar
0 votes
0 answers
20 views

How to format data for one-way intraclass correlation coefficient?

I'm testing inter-rater reliability of an instrument on a sample of subjects from several sites. Each subject is assessed by two raters, different between each site. According to this paper, I should ...
Charlie's user avatar
  • 142
2 votes
0 answers
40 views

Correct approach for evaluating correlation between multiple measurement devices used on the same subject (repeated measures)

My scenario is as follows. I am using three different devices (TempDeviceA, TempDeviceB, and TempDeviceC) to take the temperature of each animal in a group every day over several weeks. TempDeviceC is ...
SoManyQuestions's user avatar
3 votes
1 answer
49 views

Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
nahp's user avatar
  • 151
1 vote
0 answers
28 views

Extending paired t-test to compare agreement between three or more analytical instruments over time

I have three instruments that measure the concentration of dust in the air, and I want to test whether the mean difference between their measurements is zero (i.e that they give the same measurement). ...
fieldofsheep's user avatar
0 votes
0 answers
14 views

How to determine goodness-of-fit between non-parametric 2d-datasets

Lets say I have a set of paired x' and y' values and I have a N sets of reference values also consisting of paired x and y values. I would like to determine which reference set best matches by x'y' ...
James London's user avatar
1 vote
0 answers
17 views

Are agreement statistics suitable for comparing ratings produced by the same rater on different experiments?

I am interested in comparing the results of a single statistical test applied separately to two different datasets. My test produces lists of p-values and estimated effect sizes for the features of a ...
jay's user avatar
  • 143
0 votes
0 answers
31 views

What is a good metric for agreement in this problem?

Suppose I have a dataset that needs annotations. I have about $5000$ total $x$ values, and annotations by humans are needed to create $y$ values. $y$ takes a value in $\{0, 1\}$. For example, consider ...
whoisit's user avatar
  • 101
1 vote
1 answer
36 views

Estimate sensitivity and specificity from inter-rater agreement?

Assuming two binary (Y in {0, 1}) annotators or classifiers (A and B), that are: Conditionally independent, i.e. P(A=0, B=0|Y=1) = P(A=0|Y=1)*P(B=0|Y=1) and the same for Y=0. Better than random, i.e. ...
docaug's user avatar
  • 11
1 vote
0 answers
26 views

Evaluate two datasets for corresponding to given pairwise matching

Suppose I have two matched $N \times D$ datasets $X$ and $Y$. The samples are in matched order (with $n$th sample in $X$ being the same object as the $n$th sample in $Y$). The features are (or at ...
Betterthan Kwora's user avatar
1 vote
0 answers
14 views

Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
Anıl's user avatar
  • 11
2 votes
1 answer
92 views

Inter-rater reliability for mixed data types?

I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my ...
Ashti's user avatar
  • 63
3 votes
1 answer
144 views

Pearson/Spearman correlation or intraclass correlation coefficient?

I wish to study the correlation of responses of 150 participants to two concurrently-administered previously-validated questionnaires (each giving numerical summary scores as output). Both ...
user avatar
1 vote
1 answer
119 views

How to compare Gwet's ac1 coefficient between group of raters?

So I have 4 raters who scored the subjects in the study with a nominal score like: "Present", "maybe", and "absent". I made pairs of the raters and evaluated their inter-...
Kierrajames's user avatar
4 votes
3 answers
528 views

How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
Kevin's user avatar
  • 49
3 votes
1 answer
52 views

How to measure consistency of teacher grading?

I would like to determine how consistent teacher grades are for a given item, both for that teacher and across all teachers. E.g., does teacher A consistently score item 1 as a 1,2,3, or 4; and do all ...
Anthony Schmidt's user avatar
0 votes
0 answers
25 views

McNemar test or similar with multiple pairings?

Let's say I have test condition A and B, and a binary classification model. I have paired samples in conditions A and B, and I want to test if these conditions are different in classifier performance. ...
James's user avatar
  • 1
1 vote
2 answers
92 views

Testing if refusal rates are "correlated" between 3 judges

I have data looking like this: group refusal proportion (judge A) refusal proportion (judge B) refusal proportion (judge C) group 1 0.2 0.3 0.05 group 2 0.1 0.5 0.6 group 3 0.6 0.9 0.7 group 4 0....
Daniela's user avatar
  • 57
0 votes
0 answers
46 views

Accounting for occasionally correlated data in performance analysis

I'm doing a study in which a new diagnostic method is evaluated. Samples are taken from patients, both this new method and the current standard method are applied, and we calculate concordance between ...
ADdV's user avatar
  • 101
2 votes
2 answers
41 views

Estimating error rates from inter-rater reliability values

I have been tasked to approximate the error rate of a research procedure. The only data I have available are reports which specify the number of targets (n), the number of raters (k) and give pairwise ...
cactuslover's user avatar
1 vote
0 answers
52 views

Negative ICC values for inter-rater reliability

I have conducted research which assessed the technical skills of surgeons. A surgeon will perform a simulated operation, and be assessed by two different assessors whilst completing the procedure. The ...
DLCN1994's user avatar
1 vote
0 answers
45 views

Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
user21584544's user avatar
1 vote
0 answers
47 views

Inter-rater reliability of two raters for conversation analysis (multiple categories with non-binary values)

Two raters have done conversation analysis, coding all the topics in the interaction for multiple participants. There are multiple topics, but they are predetermined. There are two raters, and both ...
u-_-u's user avatar
  • 97
0 votes
0 answers
27 views

Calculating Inter-Observer Reliability when not all categories are scored

I have two main questions: 1) Is there a way to calculate IRR when the range of scores differ between observers? If not, how do you get IRR without excluding data? I am calculating inter-observer ...
user1762's user avatar
0 votes
0 answers
33 views

Preferred statistics to evaluate interrater agreement of group of raters (nominal dependent variable, two groups of raters) and compare both groups

I hypothesized that a set of symptoms is not really specific to a particular syndrome (let´s say A), so I recruited two groups of experts to classify each symptom (I have 16) to condition A, B, C (...
Dr. Fernán Arana's user avatar
1 vote
1 answer
34 views

How can I compare two methods of measuring variables for a set of objects?

I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
kate allerton's user avatar
1 vote
1 answer
91 views

Fleiss Kappa score - NaN for perfect agreement

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return ...
akalanka's user avatar
  • 111
3 votes
0 answers
51 views

Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
Yip's user avatar
  • 31
5 votes
1 answer
91 views

What's the standard deviation of the difference between randomly paired data?

Assume that we want to compare the agreement of two methods of measuring blood pressure (e.g. a machine and a sphygmomanometer). In their paper, Preiss & Fisher (2008) discuss the standard ...
COOLSerdash's user avatar
2 votes
1 answer
296 views

Any good metric for measuring multi-annotator agreement on an imbalanced dataset?

Is there an agreement method that would be well-suited for a data annotation task where: the labels are discrete classes each datapoint belongs to exactly one class (multi-class classification) each ...
Marek's user avatar
  • 143
0 votes
1 answer
30 views

Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
Novra's user avatar
  • 1
0 votes
0 answers
52 views

Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
Lukas Wallrich's user avatar
0 votes
0 answers
132 views

Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
Nelson Liu's user avatar
0 votes
0 answers
93 views

How many items for a reliable Fleiss kappa?

Suppose that I have N items that were rated by a first group of annotators on a scale from 1 to 5 (each item was rated only once). I want to compute an inter-annotator agreement score using Fleiss ...
Not Djijkjkkstra's user avatar
1 vote
1 answer
105 views

Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
crabnebul's user avatar
1 vote
0 answers
51 views

Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
Timelate's user avatar
  • 309
0 votes
0 answers
77 views

Intra-rater reliability of an instrument

I'm working on the implementation of a "new version" of an existing instrument in the posture analysis. I know that if I want to compare the measures between the old one and the new one I ...
Fede R's user avatar
  • 23
1 vote
0 answers
63 views

Inter annotator agreement with varying annotators [duplicate]

I run a human evaluation on a recruiting website. The situation is the following: There are 78 annotators. There are 100 questions. Each annotator annotates 5 questions. So, on average*, each ...
ImAUser's user avatar
  • 83
1 vote
1 answer
110 views

Pair confusion matrix for more than two cluster algorithms

How would one code the following issue: The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
Sean_TBI_Research's user avatar
0 votes
1 answer
101 views

Analyzing method comparison studies that include repeated measurements

I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison ...
kjcb's user avatar
  • 1

1
2 3 4 5
9