Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
1
vote
1answer
1k views

How to calculate inter-rater reliability with multiple raters and multiple categories per item? [duplicate]

I have a data set for which I would like to calculate the inter-rater reliability. However, this data set does not seem to fit the typical models that conventional algorithms allow for. My data set ...
2
votes
1answer
209 views

advice on a solution attempt: interrater reliability of time-point data [duplicate]

In my data, two coders annotated subjectively (but independently), when certain time-point phenomena (a specific turn in a movement pattern) occurred. The data for the first 14 seconds looks ...
3
votes
1answer
186 views

How can I calculate inter-rater reliability for continuous measures that are not normally distributed?

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
3
votes
1answer
68 views

Correlation between diagnosis codes

How do I run a correlation analysis on medical diagnosis (Dx) codes between two years at patient level. Data runs into 45000+ observations for each patient with different diseases across years and I ...
0
votes
0answers
117 views

Observer agreement in multilevel binary data

I have a set of participants indicating with drawn circles where they experience pain on a manikin of the human body. These drawings were then independently rated by two human raters: each of 29 body ...
1
vote
0answers
100 views

Is there more than one way of calculating F1-score

The reviewer of an article that I submitted to a journal commented that I never explicitly state how I calculate the F1-score. Is there more than one way to do it assuming that we are talking ...
1
vote
1answer
172 views

Comparing angle measurements from two different techniques

I'm working on a research project wherein we have 50 subjects. We have 3 x-rays per patient that all show a slightly different view of the same anatomic region. I am trying to compare an angle ...
2
votes
2answers
890 views

Inter-rater reliability for unordered categorical data

I have some content analysis data that consist of unordered mutually exclusive categories as rated by two coders. What are useful approaches towards assessing inter-rater reliability for these data? ...
5
votes
1answer
3k views

Comparing inter-rater agreement between classes of raters

I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts ...
3
votes
0answers
779 views

When to use the Kappa statistic evaluation metric?

Can someone tell me when is it appropriate to use the Kappa statistic? Also why to use it when one can use Area Under the ROC curve? Or even the Area under the precision-recall curve? So what are the ...
2
votes
0answers
402 views

Fleiss' Kappa alternative for inter-rater reliability with multiple answers

I have a dataset where researchers assign a binary yes/no answers to cells in a matrix. For each row more than one, but at least one, 'x' is allowed. For example, researchers 1 may code: ...
1
vote
0answers
54 views

A measure of correspondence between ranked ordinal data

I would like to find an appropriate way to measure the similarity between two sets of data with the following characteristics: Both sets contain 10 categorical observations. The categories observed ...
13
votes
5answers
548 views

How can I best deal with the effects of markers with differing levels of generosity in grading student papers?

Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test ...
2
votes
1answer
591 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
1
vote
0answers
326 views

Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?

I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
0
votes
1answer
190 views

What is the best way to quantify concordance of two tests?

I have two different medical diagnostic tests, both test the same condition (binary outcome). The condition in question is rather vaguely defined, so they don't always agree. What would be the best ...
5
votes
1answer
866 views

which statistic should I use for inter-rater agreement on nominal data, multiple responses

I have over 400 open-ended responses. The total has been coded by one rater, and a 10% identical sample of the total has been coded by 2 raters. There is a choice of 14 nominal codes, of which the ...
1
vote
1answer
11k views

Do I run a one-tailed or two-tailed test when running Spearman's correlation between interval and ordinal data?

I have one interval variable on a 1-25 point interval scale, i.e. placement test scores where 25 is the highest competence score (data not normally distributed), and six different ordinal variables on ...
2
votes
0answers
35 views

Analysis of ordinal data with multiple assessments per unit… the right approach?

Suppose I have a dataset with the following information: N objects, each of which can be rated The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each ...
1
vote
0answers
40 views

Expected agreement if random

I am looking to measure agreement between participants in choosing which member of a group is most like certain attributes. I want to calculate the expected agreement if they were to choose by chance. ...
1
vote
0answers
213 views

Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was ...
5
votes
1answer
2k views

Inter-rater agreement for Likert scale

What is the best inter-rater agreement test for Likert scale type questions? As far as I see, Cronbach's $\alpha$ is for internal consistency, or it shows how good items are related to describe the ...
7
votes
1answer
6k views

Creating and interpreting Bland-Altman plot

Yesterday I heard of the Bland-Altman plot for the first time. I have to compare two methods of measuring blood pressure, and I need to produce a Bland-Altman plot. I am not sure if I get everything ...
1
vote
0answers
98 views

Observer agreement with Cohen's Kappa - is there a way to combine ratings to investigate if one rating impacts the other

I am a medical scientist doing observer agreement analysis on some very simple data using Cohen's Kappa. I have 2 observers. They score a bunch of scans on a 3 point scale for a pattern called ...
0
votes
1answer
1k views

Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
0
votes
0answers
1k views

Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where T'...
-1
votes
2answers
173 views

How will two raters score videos?

I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
1
vote
1answer
200 views

How to summarize measurement variability for repeated measures?

I have a data set where different measures are taken repeatedly (3) times for multiple samples, like so: A B C X ... Y ... So, for people X, Y, etc I have ...
3
votes
0answers
250 views

How do I calculate reliability between an algorithm and multiple human experts?

Compare inter-rater reliability between 8 human experts and a computer I am running a study where I use automated analysis to analyse an EEG signal (brain activity) and quantify certain properties ...
3
votes
0answers
635 views

Inter-rater reliability: Nominal data, but *multiple ratings per unit/subject*, different raters

I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per ...
3
votes
0answers
191 views

What is the analysis techniques for comparing two sets of ratings

Say, I have collected 50 comments. For each comment, I am planning to get 10 ratings from non-expert raters for the following criteria: "interesting vs. not interesting". Based on the plurality of ...
2
votes
2answers
13k views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
5
votes
4answers
31k views

Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS. Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important). All ...
4
votes
1answer
2k views

Kendall's coefficient of concordance (W) for ratings with a lot of ties

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As ...
1
vote
0answers
122 views

Looking for agreement amongst respondents

I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important. I know beforehand that the items are related (they are a published list of ...
2
votes
0answers
136 views

New method for meta-analysis to determine agreement between multiple sources

I'm writing a paper on a method to determine the statistical significance of whether a sample of sources are in mutual agreement on specific premises. This is done by counting the number of sources in ...
6
votes
1answer
7k views

Quadratic weighted kappa versus linear weighted kappa

When should I use quadratic weighted kappa or linear weighted kappa? I have two observers evaluating the classes of a number of objects. The classes are fail, pass1, pass2, and excellent (ordinal ...
1
vote
1answer
2k views

How to evaluate concordance between 2 raters and factors that affect the concordance

I have about 1000 subjects ($id$) and each of them has some baseline variables (demographic variables $X1, X2, X3, Xn$, which can be continuous, factors). Every subject is classified by two raters ($...
2
votes
1answer
1k views

interrater reliability for multiple variables

I and another coder want to assess our agreement. Our experiment involves coding the utterances of participants. there are 17 different codes that can be assigned, and we have independently counted ...
2
votes
1answer
77 views

Macro and Micro Inter-Coder/Rater Agreement

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in https://docs.google.com/drawings/d/...
5
votes
1answer
277 views

Calculating inter rater reliability where raters and ratees only partially overlap

I'm new to inter rater reliability calculations. I have 5 developers and 6 raters. The raters ranked the first 3 of these developers based on some criteria like the following example: For example,...
2
votes
1answer
156 views

How to assess relationship between responses to multiple likert items between two raters when there are multiple rater pairs?

I have two data sets in which the same Likert scale questions were answered by two different responders. I want to calculate which level they are correlated. My data set is like ...
4
votes
1answer
241 views

How to evaluate differences of multiple Likert scale ratings of multiple items within groups?

I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in ...
1
vote
0answers
239 views

Help needed in computing Inter-rater, intra-rater variability

I have been working on a test in which I need to quantify the variability in a process. The process in question involves positioning a number of landmarks on an anatomy. There are two sets of ...
1
vote
0answers
71 views

What is most appropriate test?

I have conducted a survey using 5-Point Likert scale (Strong Agree; Agree etc.) in which a panel of experts indicated their Agreement with suggested statements. There are three sub-Groups within the ...
3
votes
1answer
238 views

What is the proper test for inter-rater reliability with three raters and ordinal ratings?

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal (...
2
votes
0answers
830 views

Which correlation analysis is appropriate for binary and ordinal variables?

Violin players were asked to rate 5 violins from least to most rich on a [0 1], 0.05-increment scale, ...
7
votes
2answers
516 views

Handling poor inter-rater reliability while minimizing the loss of data

Edit: While I do appreciate Peter Flom's suggestions and the subsequent discussion (and upvoted his answer), I am opening a bounty to solicit an answer which offers a specific, formal statistical ...
2
votes
0answers
124 views

Assessing the ability of two quantitative methods measuring the same thing

Suppose I have two scientific approaches to measuring the same physical quantity taking values between $[0,1]$. One (say method A) is generally accepted to be more accurate under certain conditions, ...
1
vote
1answer
60 views

Test for treatment change

I have a study where the goal is to see if a new diagnostic tool provides any addition to the old procedure. A set of patients have been diagnosed and classified into 7 categories, each corresponding ...