Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

0
votes
0answers
10 views

Corelation between two dichotomous variables

I have a data from a research with responses (yes/no) about three questions (A,B,C) and I would to see if the variables are correlated. First of all i compared the proportion of agreement(yes-yes, no-...
1
vote
0answers
21 views

Help analyzing data ranked on a scale

So, I thought it would be interesting to try an experiment with my family. I baked 13 different kinds cookies this year, and then told my 8 family members to rank from each cookie from 1-13. Is there ...
0
votes
0answers
36 views

weighted kappa for intra/inter-rater reliability in SPSS

I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS ...
2
votes
2answers
56 views

How to compare different clusterings?

In an 'unguided' experiment, we asked several people to make groups with a defined number of objects presented to them. They were free to do as many classes as they wanted, and to choose their names, ...
-1
votes
0answers
15 views

How i can compute Mezzich's Kappa?

Specifically, i need to compute the proportion of chance agreement in the Mezzich's Kappa formula. Thanks in advance.
1
vote
0answers
15 views

Comparing inter-rater agreement between classes of raters with only two ratings per object

In order to decide whether semi-experts can replace experts in my setting, I am interested in comparing ratings of the same objects that were done by two raters from 2 different groups (semi-expert, ...
-1
votes
0answers
40 views

Bland Altman limits of agreement vs. confidence intervals of regression parameters?

I would like to assess agreement between measurements made by two different methods. I am unsure whether to use Bland Altman analysis to establish limits of agreement, or use the confidence intervals ...
0
votes
0answers
29 views

Bayesian agreement

Assume that there are k expert measuring some quality x with a number between zero and one, $x_i \in [0,1], i=1,2,...,k$. I would like to first know the aggregated quality of x with respect to the ...
1
vote
0answers
27 views

Can I report a confidence interval for a kappa in which two raters have perfect agreement?

In a test of inter-rater agreement, I planned to report a kappa and lower+upper confidence bounds. However, I found 100% perfect agreement between the two raters. Is it possible to produce a ...
2
votes
2answers
52 views

Calculating inter-annotator agreement

Are there situations when it is allowed to omit calculating expected agreement but use only observed agreement as reliable measure? I have multi-label classification (in particular annotation of ...
0
votes
0answers
18 views

What test can show agreement between 2 large sets of repetitive data?

So basically i've measured a force that is dependend on an angle, and iv'e measured it a about 50 times. After that I changed something on the set-up and measured it again about 50 times. I checked ...
1
vote
0answers
14 views

Inter- and intra-observer agreement based on regression coefficients

I have a following study design for a group of people looking at a picture: Ordinal outcome variable - based on picture, how sure you are that A is true from 1 to 5 Binary predictor - a golden ...
0
votes
2answers
17 views

Stats test for nominal data:

I have to validate if two different methodologies yield statistically similar results to prove if we can use either of the methodologies. both the methodologies use the same sample, i.e same group of ...
1
vote
1answer
24 views

Calculate Kappa inter-rater reliability score for two categories where one has multiple but equivalent options

I'll try to be clear by describing the data and the problem. This is for a systematic review of a specific research literature. All articles can be either 'included' or 'excluded'. However for the ...
0
votes
0answers
19 views

Disagreeing measures of agreement for multilevel modeling

I created a questionnaire of 3 parts. Each part has a few questions. I now want to see if I can gather data into level 2 modeling (i.e. take each part and gather all answers to have only one number). ...
0
votes
0answers
47 views

Cohen's kappa for repeated measures longitudinal data

I have a dataset with more than 100 repeated measures of observations across multiple subjects. I want to estimate the agreement between two of the variables measured repeatedly. Note that each ...
0
votes
0answers
13 views

Inter-Rater Reliability/Agreement between “One Rater and Rest of Raters”

A snapshot of the partial rating information is given below: This example is based on the rating scale range [1-9]. The blank field indicates that ith rater has never rated the jth ratee. My first ...
0
votes
0answers
15 views

Cronbach's Alpha to Rate Videos

I would like to see if we can have reliable metrics that can be used to rate videos (V1:V15). I have 20 people for each video (id1:id20). And 2 metrics: ...
0
votes
0answers
14 views

Measures of rater agreement for timing dataset

I have a dataset that contains multiple raters. Each rater had to watch 6 videos and indicate the time at which an event occurred (e.g., 2 secs, etc.) during each trial (60 trials total). For example: ...
2
votes
0answers
257 views

How do you calculate interrater reliability for multiple items?

I have responses rated on 12 binary categories, treating the categories as separate items on the same measure. There are 10 raters, so Fleiss's Kappa seems appropriate. How would you compute ...
0
votes
0answers
13 views

Seeking a measure of multiple rater consistency for cateorical ratings with sparse data

I have multiple raters and sparse data. The raters can decide how many attributes apply to a person. I have many empty cells due to many attributes and few attributes actually chosen. What tools ...
5
votes
2answers
101 views

How to index rater agreement: multiple raters identify strenghts/weaknesses from 30 traits?

Six raters independently rated 25 people by identifying strengths and weaknesses: choosing among 35 traits (15 positive traits and 20 negative traits). The rater instructions said to identify a ...
5
votes
2answers
238 views

Compare a diagnostic test to gold standard

I have a sample of patients who underwent 2 diagnostic tests, one of which is the gold standard. I've been asked to test if the diagnosis from the experimental test is different than the one from the ...
0
votes
0answers
42 views

Calculating Kappa correlation coefficients of the data to find agreement between two raters

I am trying to calculate Kappa correlation coefficients of following data to find agreement between two raters. However, from the data set, I can see that the raters are agreeing on almost all the ...
0
votes
0answers
10 views

Measuring effectivenes of raters using Inter rater agreement

I am conducting a general survey where I get results from 5 people. When I measured the inter-rater agreement using Cohen's kappa from 4 people I got kappa value as 0.49 but when I included the 5th ...
0
votes
0answers
56 views

Internal consistency

I have to analyse a dataset in which there are drugs and used animals based on article review. The datasets looks like this: ...
1
vote
0answers
46 views

Fleiss-kappa score for interannotator agreement

In my dataset I have a set of categories, where for every category I have a set of 150 examples. Each example has been annotated as true/false by 5 human raters. I am computing the inter-annotator ...
1
vote
0answers
26 views

Intra-class correlation between two raters

I'm reading a paper that compares two raters (one human, one algorithm) that rate images with a single number (total volume of an object in the image), and there are about 100 images. Each image is an ...
1
vote
0answers
65 views

how to calculate inter-rater agreement of nominal data with multiple categories in r

coder1 results coder2 results Hi, what method should I use to calculate inter-coder reliability between these two data in r? The row stands for the cases, and the a1~ a23/A1~A23 stands for the ...
0
votes
0answers
76 views

Willmott's index of agreement range reference

Does anybody know if there is any reference that links Willmott's index of agreement (d or dr) with model performance by range, similar to Cho et al. 2012 for NSE? i.e.: 0.50 > d >0.65 --> ...
1
vote
0answers
62 views

Which scores to use when use two raters?

I am conducting an experiment in which two raters rate ideas for originality on a 6-point Likert scale. I will test the raters' interrater-reliability on 10 example ideas that both raters will do ...
0
votes
0answers
37 views

Inter-Rater Reliability (IRR) Comparison: Is it acceptable to use and average Kappas when comparing rater score to subject scores?

I am currently working on my thesis project and I have hit a bit of a snag. I have two raters who have given participants scores based upon what they wrote in open response questions. After using ...
1
vote
1answer
129 views

Degree of alignment between judges (example: equestrian judges)

When two equestrian judges are judging an equestrian event they each give their marks for every horse/rider combination in the event. Let's assume that there are 20 horses being judged. However, in ...
1
vote
0answers
111 views

Intercoder reliability for multi-label categorical data

I have a task where multiple coders can apply non-mutually exclusive categories to a list of subjects, e.g., more categories can be applied to the same subject. What could be the most appropriate ...
0
votes
0answers
39 views

What are “z-normalized scores” in the context of Fleiss' kappa coefficient?

I am tasked at work with evaluating inter-rater reliability for an experiment where speakers had to indicate on a print-out of a text which words they hear as prominent and where they hear a boundary (...
1
vote
0answers
53 views

Inter-rater Reliability for Pairwise Ranking with sparse observations

I would like to calculate inter-rater reliability for data with pairwise comparisons where different pairs are compared for different items by different raters. This doesn't seem to fit the framework ...
5
votes
1answer
218 views

Is there a specific name for this plot?

This plot represents the popularity of technologies in two "tools", vue and react. In left-top corner are specific technologies for vue but not for react, right-top technologies popular in both tools ...
2
votes
1answer
90 views

Which ICC model to use for computer vs. human agreement

I have an agreement study with the following basic construction: ~ 400 items are rated by (1) a computer algorithm, and (2) a human rater. The ratings are on a continuous scale. I would like to ...
0
votes
0answers
33 views

What is the best way to test concordance between two sources?

I have gathered RNAseq data from two different papers. They are measuring the same genes in supposedly the same population of cells and I would like a way to quantify how well they agree. No need to ...
0
votes
0answers
54 views

Krippendorff's alpha with structural zero in coincidence matrix

I'm considering using Krippendorff's alpha for inter-rater reliability statistics. Our study involves pairs of raters identifying behaviors. However, due to the way our study was designed, there are ...
0
votes
1answer
240 views

Cohen's Kappa: is it valid to average kappa for different rater pairs across multiple trials?

I'm dealing with the output of a study where raters were semi-randomly paired (based on availability) to rate a total of 50 unique but similar scenarios. Each scenario was rated by a different pair of ...
1
vote
0answers
40 views

reliability of a device for my thesis and negative icc values in spss

first time here , hoping for a great answer! so my thesis is including using a sensor to measure how far is a a person leg is from the sensor. so i measured it with a ruler and compared with sensor ...
0
votes
1answer
192 views

Which measure for inter-rater agreement for continuous data of 2 raters about multiple subjects in multiple situations?

I've considered measures like Cohen's kappa (but data is continuous), intra class correlation (reliability, not agreement), standard correlation (will be high when one rater always rates consistently ...
0
votes
1answer
136 views

Intercoder reliability without mutually-exclusive codes

I have the results from qualitatively coding the contents of ~100 images, with 7 categories and 96 tags in total. It's incredibly difficult to define mutually exclusive codes for the content we're ...
1
vote
1answer
32 views

Agreement between two machine learning models

I'm making a machine learning model (right now I'm using average weighted neural networks) to predict a binary variable. I have historical data on which I can train this model, but when new models are ...
0
votes
1answer
130 views

NaN produced from Cohen Kappa without 100% agreement

Considering the following data frame called coris_d0: ...
3
votes
0answers
46 views

Compare correlations between the same variables measured with two methods

I have measured the same variables (quantities of microorganism by 16S sequencing) on some samples in two ways, A (in the stools) and B (from an intestinal biopsy). I am trying to do two things: ...
1
vote
1answer
32 views

Which test do you suggest to use to catch the differences in paired sample

I must evaluate the goodness of an instrument according to the measures of the final products. I have two instruments: one good and the other one whose goodness I have to evaluate. The measures are ...
1
vote
1answer
157 views

what does it mean when they say some values agree within some percentage?

Maybe a silly question, as the question states, what exactly does it mean when a comparison is written in percentage? Will it be better to have let's say an agreement of 5% or 10%? In order words, ...
1
vote
0answers
734 views

How exactly is sample size determined in inter-rater reliability study?

I am conducting an inter-rater reliability study where we have four raters and the form of the rating is "Yes vs. No". Correspondingly, we chose to use Kappa value as our outcome measure. Now we want ...