Questions tagged [agreement-statistics]

Agreement is the degree to which two raters, instruments, etc, give the same value when applied to the same object. Special statistical methods have been designed for this task.

Filter by
Sorted by
Tagged with
2
votes
0answers
51 views

Measure of rater agreements for rank order?

If k raters are asked to rate the same set of objects on a continuous or Likert scale, there is the ICC3 for measuring the inter-rater agreement. Is there also an agreement measure, if all raters have ...
1
vote
0answers
15 views

Inter rater agreement where categories are not predetermined

How can I measure inter-rater agreement, e.g., using the Kappa coefficient, during an exploratory study where the categories are not predetermined?
2
votes
1answer
61 views

Inclusion of a NA category in inter-rater agreement calculation

I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not ...
0
votes
0answers
16 views

Statistical tool to compare different indices

In my research study, I have determined seven indices to measure a physical phenomenon. These indices are derived from different types of rheological tests but all measure the same phenomenon. Of ...
4
votes
1answer
25 views

Comparing two sensors timeseries that measure the same variable

I have data on one station provided by two sensors that measure solar radiation, let's call A and B where A is the more accurate and the B the less accurate one. The plot show data from one hour ...
1
vote
0answers
8 views

Is it possible to test the independence of two data sets, while taking a known error into account?

I'm doing some analysis on a device which sizes potatoes. To assess the accuracy of this device, I'm comparing (1) the data for a box of potatoes, as sized by this device, and (2) the data for the ...
0
votes
0answers
25 views

Comparing multiple arbitrary distributions

I have asked $N$ statisticians to measure a random variable $X$ based on some data $Y$. They have each returned to me a probability distribution. I want to determine whether these distributions are ...
1
vote
0answers
15 views

Fleis' Kappa statistic indicates very limited agreement between testers but ANOVA test shows very little difference between means

I have been asked to help a friend complete a statistics assigment for a non-statistics course. Disclaimer: I am not performing the work, and am merely going to explain to them how to perform the ...
2
votes
0answers
22 views

Estimating Irreducible error from kappa statistics for medical applications

In many medical applications that involve some form of classification the data is annotated by expert humans (e.g. histopathologists, radiologists). This is then taken as the ground-truth. However, it ...
0
votes
0answers
15 views

How to determine inter-rater reliability with multiple raters and multiple labels

I have a set of documents, each of which is to be labelled by any 2 of 6 raters. Each document can have more than one label. I'm trying to determine the best way to measure the level of agreement. ...
1
vote
0answers
5 views

Is a two-way repeated measures ANOVA an adequate design for this experiment?

I want to study if a continuous variables called Value changes when measured by different combinations of Raters and Methods. Since the subjects are the same in all the combinations, is a two-way ...
3
votes
1answer
40 views

Compare three sets of experimental data (Acid-Base titration)?

Using acid-base titration technique for 20 different batches of salt samples (samples are exactly same, batches are different), the residual impurities (in ppm, microgram per gram of salt) were ...
0
votes
0answers
22 views

Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
0
votes
0answers
9 views

Measures of variance for binomial distribution - “maximum” and “chance” values

I've come across a rather old textbook on statistics (Statistical Methods in Medical Research, P.Armitage, 1977) and in the section on binomial distribution there is an example that uses terminology I ...
1
vote
0answers
23 views

Which ICC model for comparing one rater vs. several raters?

I'm analysing a dataset in which a diagnostic instrument has been administered to a sample of roughly 30 patients. The instrument yields dichotomous scores (presence or absence of diagnosis) as well ...
1
vote
0answers
17 views

Finding rating for individual based on multiple raters

I am trying to find an individual agent's score based on customer scores (raters) while the customer only rates their overall experience. Context The customer enters a shop and meets a service agent ...
0
votes
0answers
9 views

Kappa inter-rater agreement for large number of raters

I have 24 students rated by 9 different professors (we have a lot of professors) ...
0
votes
0answers
9 views

Difficulties with agreement statistics and cohen's kappa

I have values of body compartments areas obtained through manual delineation using a dedicated medical software. We were two observers to perform the manual delineation and I am trying to assess the ...
1
vote
0answers
33 views

Multiple rotating adjudicators inter-rater reliability

I am analyzing data from a clinical study. In this study, each case was ranked by two to three adjudicators. The study had a panel of six adjudicators. Each case was adjudicated by two primary ...
0
votes
0answers
22 views

Test-Retest reliability with multiple raters on different subjects at different times

I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5. For example, the below data relates to 3 different "Rater"'s ...
0
votes
0answers
8 views

Correction when comparing ratings with different inter-rater reliability

I had a survey where people rated two groups of things with a number and I found that the ratings between the two groups are not significantly different (using a t-test). However, I found that the ...
0
votes
0answers
33 views

sample size for ICC and Fleiss' kappa

A study involves several continuous measurements and categorical observations (2-5 levels) from radiography images. Observations and measurements are taken by 4 or 5 readers(all readers read all ...
0
votes
1answer
34 views

Compare agreement between multiple different methods

There are 3 subjective methods to measure the severity of a condition. Each method measures it on an ordinal scale, but the scales are different: method 1 scale is 1-3 method 2 scale is 1-4 method 3 ...
1
vote
1answer
47 views

What is the number of observations required for adequate Krippendorff's alpha?

say I am working on a project to classify a number of documents. Say I have 2,000 newspaper articles that I would like to code as either containing information of lawnmowers (1) or not (0). This is a ...
0
votes
0answers
15 views

Sample size cutoff for agreement coefficients

New poster here. Would love some help thinking through this. Suppose I am observing individuals in public space, and I am recording the number of occurrences of a target behavior (e.g., looking at a ...
0
votes
0answers
30 views

How to calculate sample size estimation for fleiss kappa?

We have six categories that raters have to choose from. We have an intervention that should improve the rating quality. In previous studies the interobserver agreement is fleiss kappa 0.43 We expect ...
0
votes
0answers
10 views

Metric for evaluating agreement at inter-rater reliability for a single subject by multiple raters

I'm making a rating survey in R (Shiny) and I'm tryng to find a metric that can evaluate the agreement but for only one of the "questions" in the survey. The ratings range from 1 to 5. There ...
1
vote
0answers
18 views

Determining whether a rating scale is ordinal or nominal from data?

I am occasionally presented with data where a rating scale was used (i.e. options 1 through 5). Sometimes these are ordinal, e.g. Very positive. Somewhat positive Neutral Somewhat negative Very ...
0
votes
0answers
19 views

Help Performing Inter-Rater(Coder) Reliability in NVivo

I'm a student using the software NVivo 12 for Windows to perform qualitative coding on interview transcripts. One other person will be coding the same interviews independently so that we can perform ...
1
vote
2answers
34 views

Inter-rater agreement after rater 1-based sampling

Rater 1 rated (+/-) some cases. Rater 2 gets to rate all the Rater 1 + cases, but only half of the rater 1 - cases. Is there a way to account for this sampling scheme when calculating inter-rater ...
0
votes
0answers
9 views

What is the Corrected Pearson Correlation?

In an article about Inter-Rater Reliability, they reported on following 2 statistics: Pearson’s product-moment coefficients (r) and corrected Pearson’s product-moment coefficients (corrected r) ...
0
votes
0answers
20 views

Std error of measurement and CIs for inter-rater observations

Reliability is often reported somewhat abstractly and can be difficult to interpret (e.g., ICC = 0.9). For test-retest reliability, it can be helpful to express measurement error in the scale of the ...
1
vote
0answers
65 views

Inter-rater agreement (raters >2) of non-random items

I ran a survey where participants (P) had to choose one category out of six (C1-C6) for each of five items. The data looks like this: ...
0
votes
1answer
31 views

Repeated measures ANOVA for equality of two measure

I have used two different designs for a questionnaire to measure the same variable, and I want to investigate if changing the design led to a different measurement as a result. I have a within-...
0
votes
0answers
13 views

How to compare inter-rater reliability between two time points?

We have two sets of raters judge the same items before and after a training, resulting in an ICC and kappa value for each time point (pre- and post-training). How would you statistically compare pre- ...
0
votes
0answers
20 views

Difference between Lin's and Kendall's concordance coefficient

I am trying to decide whether to use Lin's and Kendall's concordance coefficient to analyse agreement in 2 ratings of non-normal data (some of these are non-normal continuous variables and some are ...
1
vote
2answers
34 views

Fantasy Football Expert Consensus Rankings

Suppose there are n experts who rank m players. Let's assume there are no ties, a lower ranking is considered "better", and the rankings done by each expert are complete. For each player, we know: ...
1
vote
0answers
37 views

How to use Cronbach's Alpha?

Is it appropriate to use Cronbach's Alpha for nominal variables? I have 20 nominal variables (two dimensions with 10 questions each) that ask about the characteristics of work performance. The ...
0
votes
0answers
37 views

sample size calculation for intraclass correlation coefficient (ICC)

I want to design a study when I'll be comparing three measurements of a variable in imaging and then compute an ICC as a measure of agreement since I'll be obtain a continuous measurement. I read ...
1
vote
1answer
11 views

Principled Method for Reweighting Labels Based on Labeler Consistency?

I have a large dataset where each datum has been labeled by multiple raters, and each rater has labeled multiple datums. Rather than assigning to each datum the naive average of all of the labels ...
0
votes
0answers
20 views

Which metric for inter-annotator agreement of ranked independent items?

I'm looking for a suitable metric that measures the agreement between many annotators m (say, 15) annotating a set of independent product reviews ...
0
votes
0answers
20 views

Multi-class inter-annotator agreement for overlapping time windows of annotations

My work involves a time series data of sensor measurements. I have a tool to annotate this dataset that allows annotators to flexibly add labels to varying time-windows. I have two annotators who have ...
0
votes
0answers
45 views

Comparing an individual response (inter and intra groups) on likert scale ..best method to use

We are looking to identify the best (focus is more on pragmatically workable method) statistic to use to do the following: We have survey data where say 100 employees (belonging to 3 groups - ...
0
votes
0answers
13 views

Draw meaning from Kendall's W

I have calculated the Kendall W value for a survey I made with 16 judges (m = 16) and 20 ranking questions (n = 20) ranging from ...
0
votes
0answers
17 views

Investigator efficiency computation

I need a little help with inferring ground truth and investigation capabilities from data. I have an incomplete matrix consisting of binary decisions taken by investigators on documents i.e. for every ...
0
votes
1answer
225 views

Cohen's Kappa for more than two categories

I have data set of teacher's evaluation rated by 4 different raters. The teacher's are evaluated for 13 different categories for example (Interaction with students, lesson delivery etc). All of the 4 ...
0
votes
2answers
30 views

Which is the best coefficient to compute a “user agreement” metric?

Let us suppose that N users are asked to answer yes/no to M questions. For example: ...
0
votes
0answers
22 views

Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
2
votes
1answer
122 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...

1
2 3 4 5
7