Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Inter-Rater Reliability for Relative Strength of a Measure

Let's start with a classic: You're trying to get 3 judges to measure the creativity of different art projects, from 1-10, with 1 being 'Uninspired' and 10 being 'Incredibly Creative'. You want to ...
1
vote
0answers
39 views

Inter-rater agreement for non-fully crossed designs

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and ...
3
votes
0answers
28 views

Inter-rater agreement: number of raters vs. number of test subjects

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical ...
0
votes
0answers
28 views

Krippendorff's alpha (and other IRR stats) giving counter-intuitive results

I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3. I need a measure of inter-rater reliability that ...
0
votes
0answers
8 views

How to Compute Krippendorff's Alpha for Text Continua?

I have a question about the inter-annotator reliability measure of Krippendorff's alpha for text continua or text unitization. The measure is explained in detail in the paper. I have texts and 2 ...
0
votes
0answers
18 views

Krippendorff's Alpha for unitizing texts

I have a question about using alpha to measure Krippendorff's inter-annotator reliability in qualitative data. After reading the papers, I'm still not sure how to handle this case. I'm working on a ...
1
vote
0answers
49 views

Krippendorff's alpha in R for multi-label annotation

I need to calculate inter-rater agreement (2 annotators) on a small dataset (let's say 10 items), where annotators used between 1..4 labels per item. The total number of different labels is pretty ...
0
votes
0answers
53 views

nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?

I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
3
votes
1answer
153 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
1
vote
1answer
11 views

Is it appropriate to use a confusion matrix for frequency data?

I am testing the accuracy of a machine learning approach that counts cars in images. I have both a predicted dataset and a "real" dataset that was generated by a human. For example, this is what my ...
2
votes
0answers
16 views

Applying inter-rater reliability principles

Background: I want to perform a survey/questionnaire for a group of people with very different background and knowledge. Some of them are environmental scientists, anthropologists, and climate ...
0
votes
0answers
18 views

Krippendorff's alpha getting low after restricting data

I am calculating inter-rater agreement for ~350 raters, each of who rated 100 items using Likert scale. The original scale has range [-3,3], using this python implementation, I got $\alpha=.35$, which ...
0
votes
0answers
10 views

Inter-rater reliability many coders but only one case

I have a specific performance assessment metric which I aim to get some reliability and validity data for. And I really would appreciate some help for the reliability: Background In my research, ...
0
votes
0answers
35 views

Bootstrapping non-random samples

I perform some analysis (with the goal to select some relevant variables that are related between two experiments) on a group of samples (n = 158) but I would like to know how robust are, before going ...
2
votes
0answers
9 views

estimation inter rater reliability for strings of characters (i.e. URLs)

I have multiple raters extracting URLs from the internet based on search terms. The core issue is that a URL amounts to a string wherein two raters might come to the same URL but one string is a ...
1
vote
0answers
7 views

Demonstrating improved agreement on clinical decision making using the Kappa coefficient

I have a large number of respiratory doctors - some from University hospitals and some from community non-academic hospitals. They have been asked to evaluate 60 patients with various different lung ...
1
vote
0answers
22 views

Comparing 3 raster maps of continuous surface data in R

I have 3 maps representing spatial distribution in hunting pressure. These maps were derived using different methods and I am now interested in comparing them to assess how they might differ/agree ...
1
vote
1answer
37 views

Are there any statistical tests or measures for paired data with more than two groups?

Are there any statistical tests or measures for paired data with more than two groups? My dependent variable is pretty flexible, but I'd really like to have five independent variables compared ...
0
votes
0answers
7 views

Measuring inter-rater reliability in a manual clustering task

There are plenty of methods for measuring inter-rater reliability in tasks involving rankings, classifications, and labeling (joint probability, Kappa statistics, etc.) There doesn't however seem to ...
0
votes
0answers
9 views

Best way to measure reliability of auto-correlated data

I'm trying to compare different ambulatory devices for measuring heart rate against a gold standard in-lab assessment. What is the best way to determine whether a device is worth using. I can use ...
2
votes
1answer
23 views

Cohen's Kappa - extreme disagreement doesn't register as strongly as extreme agreement

I'm wondering why, for the following data, the Kappa statistic wasn't $\pm 1$ but instead $-0.95$ for complete disagreement and $1$ for complete agreement. I expected them to have the same value. ...
1
vote
1answer
55 views

interrater-reliability with likert-scale and different raters

I am collecting data from a survey that includes a likert scale of 5 points. Each rater answers 18 questions (probably different questions). For each question I will have answers from 10 different ...
0
votes
0answers
18 views

intra rater variability

150 raters from 30 different centres inside 10 geographical region (so 3 centres from 10 regions and raters from each centre) grade 10 images twice on an ordinal scale. How can I calculate intra ...
0
votes
0answers
193 views

Corelation between two dichotomous variables

I have a data from a research with responses (yes/no) about three questions (A,B,C) and I would to see if the variables are correlated. First of all i compared the proportion of agreement(yes-yes, no-...
1
vote
0answers
28 views

Help analyzing data ranked on a scale

So, I thought it would be interesting to try an experiment with my family. I baked 13 different kinds cookies this year, and then told my 8 family members to rank from each cookie from 1-13. Is there ...
0
votes
0answers
302 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
3
votes
3answers
75 views

How to compare different clusterings?

In an 'unguided' experiment, we asked several people to make groups with a defined number of objects presented to them. They were free to do as many classes as they wanted, and to choose their names, ...
1
vote
0answers
19 views

Comparing inter-rater agreement between classes of raters with only two ratings per object

In order to decide whether semi-experts can replace experts in my setting, I am interested in comparing ratings of the same objects that were done by two raters from 2 different groups (semi-expert, ...
0
votes
0answers
39 views

Bayesian agreement

Assume that there are k expert measuring some quality x with a number between zero and one, $x_i \in [0,1], i=1,2,...,k$. I would like to first know the aggregated quality of x with respect to the ...
1
vote
0answers
42 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
2
votes
2answers
92 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
1
vote
0answers
23 views

Inter- and intra-observer agreement based on regression coefficients

I have a following study design for a group of people looking at a picture: Ordinal outcome variable - based on picture, how sure you are that A is true from 1 to 5 Binary predictor - a golden ...
0
votes
2answers
35 views

Stats test for nominal data:

I have to validate if two different methodologies yield statistically similar results to prove if we can use either of the methodologies. both the methodologies use the same sample, i.e same group of ...
1
vote
1answer
53 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
0
votes
0answers
47 views

Disagreeing measures of agreement for multilevel modeling

I created a questionnaire of 3 parts. Each part has a few questions. I now want to see if I can gather data into level 2 modeling (i.e. take each part and gather all answers to have only one number). ...
0
votes
0answers
79 views

Cohen's kappa for repeated measures longitudinal data

I have a dataset with more than 100 repeated measures of observations across multiple subjects. I want to estimate the agreement between two of the variables measured repeatedly. Note that each ...
0
votes
0answers
30 views

Inter-Rater Reliability/Agreement between “One Rater and Rest of Raters”

A snapshot of the partial rating information is given below: This example is based on the rating scale range [1-9]. The blank field indicates that ith rater has never rated the jth ratee. My first ...
3
votes
0answers
490 views

How do you calculate interrater reliability for multiple items?

I have responses rated on 12 binary categories, treating the categories as separate items on the same measure. There are 10 raters, so Fleiss's Kappa seems appropriate. How would you compute ...
5
votes
2answers
125 views

How to index rater agreement: multiple raters identify strenghts/weaknesses from 30 traits?

Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a ...
5
votes
2answers
697 views

Compare a diagnostic test to gold standard

I have a sample of patients who underwent 2 diagnostic tests, one of which is the gold standard. I've been asked to test if the diagnosis from the experimental test is different than the one from the ...
0
votes
0answers
58 views

Internal consistency

I have to analyse a dataset in which there are drugs and used animals based on article review. The datasets looks like this: ...
1
vote
0answers
119 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
35 views

Intra-class correlation between two raters

I'm reading a paper that compares two raters (one human, one algorithm) that rate images with a single number (total volume of an object in the image), and there are about 100 images. Each image is an ...
1
vote
0answers
106 views

how to calculate inter-rater agreement of nominal data with multiple categories in r

coder1 results coder2 results Hi, what method should I use to calculate inter-coder reliability between these two data in r? The row stands for the cases, and the a1~ a23/A1~A23 stands for the ...
1
vote
0answers
70 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
1
vote
1answer
135 views

Degree of alignment between judges (example: equestrian judges)

When two equestrian judges are judging an equestrian event they each give their marks for every horse/rider combination in the event. Let's assume that there are 20 horses being judged. However, in ...
2
votes
0answers
159 views

Intercoder reliability for multi-label categorical data

I have a task where multiple coders can apply non-mutually exclusive categories to a list of subjects, e.g., more categories can be applied to the same subject. What could be the most appropriate ...
1
vote
0answers
70 views

Inter-rater Reliability for Pairwise Ranking with sparse observations

I would like to calculate inter-rater reliability for data with pairwise comparisons where different pairs are compared for different items by different raters. This doesn't seem to fit the framework ...
5
votes
1answer
258 views

Is there a specific name for this plot?

This plot represents the popularity of technologies in two "tools", vue and react. In left-top corner are specific technologies for vue but not for react, right-top technologies popular in both tools ...
2
votes
1answer
121 views

Which ICC model to use for computer vs. human agreement

I have an agreement study with the following basic construction: ~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to ...