Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
1 vote
0 answers
20 views

Proof of lower bound of Cohen's kappa

Can anyone refer me to a paper or book showing that Cohen's kappa is bounded below by $-1$? I've read various papers stating this, but I have never found a complete proof. This question was asked here,...
nahp's user avatar
  • 111
1 vote
0 answers
25 views

Extending paired t-test to compare agreement between three or more analytical instruments over time

I have three instruments that measure the concentration of dust in the air, and I want to test whether the mean difference between their measurements is zero (i.e that they give the same measurement). ...
fieldofsheep's user avatar
0 votes
0 answers
11 views

How to determine goodness-of-fit between non-parametric 2d-datasets

Lets say I have a set of paired x' and y' values and I have a N sets of reference values also consisting of paired x and y values. I would like to determine which reference set best matches by x'y' ...
James London's user avatar
1 vote
0 answers
14 views

Are agreement statistics suitable for comparing ratings produced by the same rater on different experiments?

I am interested in comparing the results of a single statistical test applied separately to two different datasets. My test produces lists of p-values and estimated effect sizes for the features of a ...
jay's user avatar
  • 143
0 votes
0 answers
29 views

What is a good metric for agreement in this problem?

Suppose I have a dataset that needs annotations. I have about $5000$ total $x$ values, and annotations by humans are needed to create $y$ values. $y$ takes a value in $\{0, 1\}$. For example, consider ...
whoisit's user avatar
  • 101
1 vote
1 answer
31 views

Estimate sensitivity and specificity from inter-rater agreement?

Assuming two binary (Y in {0, 1}) annotators or classifiers (A and B), that are: Conditionally independent, i.e. P(A=0, B=0|Y=1) = P(A=0|Y=1)*P(B=0|Y=1) and the same for Y=0. Better than random, i.e. ...
docaug's user avatar
  • 11
1 vote
0 answers
24 views

Evaluate two datasets for corresponding to given pairwise matching

Suppose I have two matched $N \times D$ datasets $X$ and $Y$. The samples are in matched order (with $n$th sample in $X$ being the same object as the $n$th sample in $Y$). The features are (or at ...
Betterthan Kwora's user avatar
1 vote
0 answers
12 views

Calculation of Cohen's Kappa with different categories and number of instances

I have a dataset that contains commentary on football matches. We have two labelers, and they have labeled which team is attacking in each commentary. As you can see, different matches have different ...
Anıl's user avatar
  • 11
2 votes
1 answer
87 views

Inter-rater reliability for mixed data types?

I am very new to statistics, and I need help determining the the best inter-reliability test for a questionnaire. My questionnaire will be assessed by three raters. I'm just providing the sample of my ...
Ashti's user avatar
  • 63
3 votes
1 answer
101 views

Pearson/Spearman correlation or intraclass correlation coefficient?

I wish to study the correlation of responses of 150 participants to two concurrently-administered previously-validated questionnaires (each giving numerical summary scores as output). Both ...
user avatar
1 vote
1 answer
66 views

How to compare Gwet's ac1 coefficient between group of raters?

So I have 4 raters who scored the subjects in the study with a nominal score like: "Present", "maybe", and "absent". I made pairs of the raters and evaluated their inter-...
Kierrajames's user avatar
4 votes
2 answers
400 views

How to calculate sample size for measuring level of agreement between devices?

I am doing a study to measure the level of agreement between two pieces of medical equipment (one gold standard one and a new device). I will measure the agreement using weighted Cohen's kappa and ...
Kevin's user avatar
  • 49
3 votes
1 answer
46 views

How to measure consistency of teacher grading?

I would like to determine how consistent teacher grades are for a given item, both for that teacher and across all teachers. E.g., does teacher A consistently score item 1 as a 1,2,3, or 4; and do all ...
Anthony Schmidt's user avatar
0 votes
0 answers
13 views

McNemar test or similar with multiple pairings?

Let's say I have test condition A and B, and a binary classification model. I have paired samples in conditions A and B, and I want to test if these conditions are different in classifier performance. ...
James's user avatar
  • 1
1 vote
2 answers
91 views

Testing if refusal rates are "correlated" between 3 judges

I have data looking like this: group refusal proportion (judge A) refusal proportion (judge B) refusal proportion (judge C) group 1 0.2 0.3 0.05 group 2 0.1 0.5 0.6 group 3 0.6 0.9 0.7 group 4 0....
Daniela's user avatar
  • 57
0 votes
0 answers
46 views

Accounting for occasionally correlated data in performance analysis

I'm doing a study in which a new diagnostic method is evaluated. Samples are taken from patients, both this new method and the current standard method are applied, and we calculate concordance between ...
ADdV's user avatar
  • 101
2 votes
2 answers
39 views

Estimating error rates from inter-rater reliability values

I have been tasked to approximate the error rate of a research procedure. The only data I have available are reports which specify the number of targets (n), the number of raters (k) and give pairwise ...
cactuslover's user avatar
1 vote
0 answers
42 views

Negative ICC values for inter-rater reliability

I have conducted research which assessed the technical skills of surgeons. A surgeon will perform a simulated operation, and be assessed by two different assessors whilst completing the procedure. The ...
DLCN1994's user avatar
1 vote
0 answers
33 views

Confused on Cohen's Kappa - Exploring Rater Agreement Stats

I am interested in conducting a survey where I show raters a graph then ask for their thoughts. I will do this by providing statement for which the raters will select a value on a 5 point likert scale ...
user21584544's user avatar
1 vote
0 answers
42 views

Inter-rater reliability of two raters for conversation analysis (multiple categories with non-binary values)

Two raters have done conversation analysis, coding all the topics in the interaction for multiple participants. There are multiple topics, but they are predetermined. There are two raters, and both ...
u-_-u's user avatar
  • 97
0 votes
0 answers
19 views

Calculating Inter-Observer Reliability when not all categories are scored

I have two main questions: 1) Is there a way to calculate IRR when the range of scores differ between observers? If not, how do you get IRR without excluding data? I am calculating inter-observer ...
user1762's user avatar
0 votes
0 answers
30 views

Preferred statistics to evaluate interrater agreement of group of raters (nominal dependent variable, two groups of raters) and compare both groups

I hypothesized that a set of symptoms is not really specific to a particular syndrome (let´s say A), so I recruited two groups of experts to classify each symptom (I have 16) to condition A, B, C (...
Dr. Fernán Arana's user avatar
1 vote
1 answer
34 views

How can I compare two methods of measuring variables for a set of objects?

I have a set of objects for which I measured the variables in two different ways. How can I determine whether these datasets overlap well in the feature space? I was thinking about calculating some ...
kate allerton's user avatar
1 vote
1 answer
65 views

Fleiss Kappa score - NaN for perfect agreement

I am working on a dataset that has three raters. Rating are Yes/No. I have a set of ratings where all raters said No. I used a R package and excel formulas to calculate the kappa score. Both return ...
akalanka's user avatar
  • 111
3 votes
0 answers
39 views

Difference between kappa statistics

I am currently developing an instrument and have some questions on the use of kappa statistics for interrater agreement used for content validity. I have gone through previous literature and I still ...
Yip's user avatar
  • 31
0 votes
0 answers
9 views

Inter-rater reliability against multiple raters

I am conducting inter-rater reliability by doing a clinical test of 25 participants. Can I do the reliability against multiple raters? I mean that I will do the test for all the participants and then ...
aya khabir's user avatar
0 votes
0 answers
16 views

Can you use inter-rater reliability to measure reliability of a whole measure with items measuring different things? (Krippendorff's Alpha or similar)

Let's say that you have one case study, where 30 raters are asked to apply a measure that has 5 items. Can you compute a single Krippendorff's Alpha (kalpha) value across all 5 items (that are the ...
jstat's user avatar
  • 1
0 votes
0 answers
86 views

Calculate minimum sample size for Fleiss Kappa

Can you please explain to me how to calculate a minimum sample size (rated subjects) for Fleiss Kappa? I cannot find neither calculators to do this, nor some reference values. Thank you very much. G
Giulia's user avatar
  • 1
5 votes
1 answer
77 views

What's the standard deviation of the difference between randomly paired data?

Assume that we want to compare the agreement of two methods of measuring blood pressure (e.g. a machine and a sphygmomanometer). In their paper, Preiss & Fisher (2008) discuss the standard ...
COOLSerdash's user avatar
  • 30.1k
2 votes
1 answer
186 views

Any good metric for measuring multi-annotator agreement on an imbalanced dataset?

Is there an agreement method that would be well-suited for a data annotation task where: the labels are discrete classes each datapoint belongs to exactly one class (multi-class classification) each ...
Marek's user avatar
  • 143
0 votes
0 answers
22 views

Calculate Fleiss Kappa with annotator replacement

If three annotators, A, B, and C, label 180 data, In the middle of the labeling process, C is replaced with D, and A, B, and D label 200 data. Can it count and average for three labelers as whole 380 ...
Novra's user avatar
  • 1
0 votes
0 answers
23 views

Can you use Krippendorff's Alpha to assess reliability of an entire measure using one case study?

I am interested in using Krippendorff's Alpha for a dataset containing a large number of raters to assess the reliability of using an assessment tool for case studies, which contains 8 items with ...
jstat's user avatar
  • 1
0 votes
0 answers
43 views

Asymmetric confidence intervals for Cohen's kappa

I am trying to understand sample sizes required to estimate Cohen's k with a given precision. I am aware that the traditional confidence intervals do not achieve the nominal coverage in small samples, ...
Lukas Wallrich's user avatar
0 votes
0 answers
38 views

Is there such thing as inter-annotator agreement for a set of rankings?

I'm familiar with inter-annotator agreement (or inter-rater reliability) metrics for data that has categorical annotations, but what about a set of data samples that are ranked by several annotators? ...
Sean's user avatar
  • 3,787
0 votes
0 answers
82 views

What is the correct Kappa coefficient to use when more than two labelers are present?

I have a dataset consisting of m rows and 2 columns. The m rows denote a patient and the columns denote the label assigned to them. The label size can be arbitrarily long, with labelers given the ...
desert_ranger's user avatar
0 votes
0 answers
16 views

Agreement statistic for likert scale evaluation of different image modalities

I have some data of dental images. The same image was taken using the gold standard method and using an alternative new more cost effective method. Evaluators rate each image (blinded) based on a ...
user213352's user avatar
0 votes
0 answers
120 views

Calculating Cohen's Kappa for a dataset when categories differ between examples

Cohen's Kappa is a measure of inter-annotator agreement that is traditionally applied to a setting where you have two raters who each classify $N$ items into $C$ mutually exclusive categories. For ...
Nelson Liu's user avatar
0 votes
0 answers
85 views

How many items for a reliable Fleiss kappa?

Suppose that I have N items that were rated by a first group of annotators on a scale from 1 to 5 (each item was rated only once). I want to compute an inter-annotator agreement score using Fleiss ...
Not Djijkjkkstra's user avatar
1 vote
1 answer
84 views

Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $p_j$ is defined as the proportion of ratings which were to ...
crabnebul's user avatar
1 vote
0 answers
42 views

Kappa coefficient, prevalence and bias. How to interpret wide confidence intervals, some falling outside [-1,1]

I an assessing the agreement between two measurements methods on a binary presence/absence metric. Here is one example of a contingency table: ...
Timelate's user avatar
  • 309
0 votes
0 answers
73 views

Intra-rater reliability of an instrument

I'm working on the implementation of a "new version" of an existing instrument in the posture analysis. I know that if I want to compare the measures between the old one and the new one I ...
Fede R's user avatar
  • 23
1 vote
0 answers
61 views

Inter annotator agreement with varying annotators [duplicate]

I run a human evaluation on a recruiting website. The situation is the following: There are 78 annotators. There are 100 questions. Each annotator annotates 5 questions. So, on average*, each ...
ImAUser's user avatar
  • 83
1 vote
1 answer
98 views

Pair confusion matrix for more than two cluster algorithms

How would one code the following issue: The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
Sean_TBI_Research's user avatar
0 votes
1 answer
70 views

Analyzing method comparison studies that include repeated measurements

I am currently a graduate student working on my thesis, trying to analyze method comparison studies that include repeated measurements. I have a dataset created in the context of a method comparison ...
kjcb's user avatar
  • 1
1 vote
1 answer
83 views

Should we test only 20% of the sample for intraclass correlation coefficient?

I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post ...
Larissa Cury's user avatar
2 votes
1 answer
291 views

inter-rater reliability for numeric variables

I've seen some questions on inter-rater reliability here for categorical variables. I'm wondering what would be an adequate approach to test inter-rater reliability between two raters for a numeric ...
Larissa Cury's user avatar
1 vote
1 answer
37 views

Inter-rater reliability (complex survey )

Help calculating Inter-rater reliability in a complex online survey (free text). The questions are free text. We have now coded the potential answers. Examples: 1.) Hobbies (Time in nature, mind-body ...
Vit Frac's user avatar
1 vote
1 answer
315 views

Reporting results of random seeds and bootstrapping (stats)

I am trying to compare my model's (a neural network) results with human-scored results on the same type of data. However, there are no ground truth labels (humans disagree on classifying these results)...
Adam Jones's user avatar
6 votes
1 answer
1k views

Measures to compare clustering partitions

What are the most used measures (coefficients) to compare two partitions of objects into clusters? I am speaking of validating the results of clustering, not of classifying; the measures known as ...
ttnphns's user avatar
  • 57.2k
4 votes
1 answer
449 views

Measures to compare classification partitions

What are the most used measures (coefficients) to compare two partitions of objects into classes? I am speaking of validating the results of classification, not of clustering; the measures known as ...
ttnphns's user avatar
  • 57.2k

1
2 3 4 5
9