Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
0
votes
0answers
30 views

Interpretation of Bland-Altman Plot for measuring temperature differences

I have 3 temperature sensors namely, A, B, and C. The data I have is hourly reading for around 2 year period. Now, I am trying to see that the temperature readings are in agreement (because they are ...
0
votes
0answers
33 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
15 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
1
vote
1answer
17 views

Cohen's kappa and confounders

I'm interested in how certain factors affect the inter-rater reliability of two driving examiners (sitting in the back seats) in deciding whether to pass or fail the examinee. For example, age, ...
0
votes
2answers
19 views

Chi square test between samples with very different size

I am to estimate the chi square between two samples of size N and M, where in general N << M (e.g. N=100,000 M=10,000,000). In turn, this affects the uncertainty of each sample. I wonder if ...
0
votes
0answers
14 views

Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
0
votes
0answers
9 views

Inter-Rater Reliability for Relative Strength of a Measure

Let's start with a classic: You're trying to get 3 judges to measure the creativity of different art projects, from 1-10, with 1 being 'Uninspired' and 10 being 'Incredibly Creative'. You want to ...
1
vote
0answers
48 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
0answers
32 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
0
votes
0answers
28 views

Krippendorff's alpha (and other IRR stats) giving counter-intuitive results

I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3. I need a measure of inter-rater reliability that ...
0
votes
0answers
14 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
0answers
21 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
1
vote
0answers
75 views

Krippendorff's alpha in R for multi-label annotation

I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1..4 labels per item. The total number of different labels is pretty ...
0
votes
0answers
100 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
3
votes
1answer
204 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
1
vote
1answer
18 views

Is it appropriate to use a confusion matrix for frequency data?

I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
2
votes
0answers
20 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
0
votes
0answers
21 views

Krippendorff's alpha getting low after restricting data

I am calculating inter-rater agreement for ~350 raters, each of who rated 100 items using Likert scale. The original scale has range [-3,3], using this python implementation, I got $\alpha=.35$, which ...
0
votes
0answers
10 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
0
votes
0answers
45 views

Bootstrapping non-random samples

I perform some analysis (with the goal to select some relevant variables that are related between two experiments) on a group of samples (n = 158) but I would like to know how robust are, before going ...
2
votes
0answers
9 views

estimation inter rater reliability for strings of characters (i.e. URLs)

I have multiple raters extracting URLs from the internet based on search terms. The core issue is that a URL amounts to a string wherein two raters might come to the same URL but one string is a ...
1
vote
0answers
9 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
1
vote
0answers
23 views

Comparing 3 raster maps of continuous surface data in R

I have 3 maps representing spatial distribution in hunting pressure. These maps were derived using different methods and I am now interested in comparing them to assess how they might differ/agree ...
1
vote
1answer
41 views

Are there any statistical tests or measures for paired data with more than two groups?

Are there any statistical tests or measures for paired data with more than two groups? My dependent variable is pretty flexible, but I'd really like to have five independent variables compared ...
0
votes
0answers
7 views

Measuring inter-rater reliability in a manual clustering task

There are plenty of methods for measuring inter-rater reliability in tasks involving rankings, classifications, and labeling (joint probability, Kappa statistics, etc.) There doesn't however seem to ...
0
votes
0answers
9 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
2
votes
1answer
24 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
1answer
73 views

interrater-reliability with likert-scale and different raters

I am collecting data from a survey that includes a likert scale of 5 points. Each rater answers 18 questions (probably different questions). For each question I will have answers from 10 different ...
0
votes
0answers
18 views

intra rater variability

150 raters from 30 different centres inside 10 geographical region (so 3 centres from 10 regions and raters from each centre) grade 10 images twice on an ordinal scale. How can I calculate intra ...
0
votes
0answers
223 views

Corelation between two dichotomous variables

I have a data from a research with responses (yes/no) about three questions (A,B,C) and I would to see if the variables are correlated. First of all i compared the proportion of agreement(yes-yes, no-...
1
vote
0answers
28 views

Help analyzing data ranked on a scale

So, I thought it would be interesting to try an experiment with my family. I baked 13 different kinds cookies this year, and then told my 8 family members to rank from each cookie from 1-13. Is there ...
0
votes
0answers
388 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
3
votes
3answers
77 views

How to compare different clusterings?

In an 'unguided' experiment, we asked several people to make groups with a defined number of objects presented to them. They were free to do as many classes as they wanted, and to choose their names, ...
1
vote
0answers
19 views

Comparing inter-rater agreement between classes of raters with only two ratings per object

In order to decide whether semi-experts can replace experts in my setting, I am interested in comparing ratings of the same objects that were done by two raters from 2 different groups (semi-expert, ...
0
votes
0answers
42 views

Bayesian agreement

Assume that there are k expert measuring some quality x with a number between zero and one, $x_i \in [0,1], i=1,2,...,k$. I would like to first know the aggregated quality of x with respect to the ...
1
vote
0answers
47 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
2
votes
2answers
97 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
1
vote
0answers
23 views

Inter- and intra-observer agreement based on regression coefficients

I have a following study design for a group of people looking at a picture: Ordinal outcome variable - based on picture, how sure you are that A is true from 1 to 5 Binary predictor - a golden ...
0
votes
2answers
38 views

Stats test for nominal data:

I have to validate if two different methodologies yield statistically similar results to prove if we can use either of the methodologies. both the methodologies use the same sample, i.e same group of ...
1
vote
1answer
58 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
0
votes
0answers
33 views

Inter-Rater Reliability/Agreement between “One Rater and Rest of Raters”

A snapshot of the partial rating information is given below: This example is based on the rating scale range [1-9]. The blank field indicates that ith rater has never rated the jth ratee. My first ...
3
votes
0answers
564 views

How do you calculate interrater reliability for multiple items?

I have responses rated on 12 binary categories, treating the categories as separate items on the same measure. There are 10 raters, so Fleiss's Kappa seems appropriate. How would you compute ...
6
votes
2answers
130 views

How to index rater agreement: multiple raters identify strenghts/weaknesses from 30 traits?

Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a ...
5
votes
2answers
800 views

Compare a diagnostic test to gold standard

I have a sample of patients who underwent 2 diagnostic tests, one of which is the gold standard. I've been asked to test if the diagnosis from the experimental test is different than the one from the ...
0
votes
0answers
60 views

Internal consistency

I have to analyse a dataset in which there are drugs and used animals based on article review. The datasets looks like this: ...
1
vote
0answers
129 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
36 views

Intra-class correlation between two raters

I'm reading a paper that compares two raters (one human, one algorithm) that rate images with a single number (total volume of an object in the image), and there are about 100 images. Each image is an ...
1
vote
0answers
109 views

how to calculate inter-rater agreement of nominal data with multiple categories in r

coder1 results coder2 results Hi, what method should I use to calculate inter-coder reliability between these two data in r? The row stands for the cases, and the a1~ a23/A1~A23 stands for the ...
1
vote
0answers
73 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
1
vote
1answer
135 views

Degree of alignment between judges (example: equestrian judges)

When two equestrian judges are judging an equestrian event they each give their marks for every horse/rider combination in the event. Let's assume that there are 20 horses being judged. However, in ...