Jeffrey Girard
  • Member for 5 years, 8 months
  • Last seen this week
1 answers
1 votes
336 views
How can I calculate inter-rater agreement for two raters assigning ranks?
1 votes

You can use Kendall's tau for two raters or Kendall's W for more than two raters. Due to its intuitive appeal, I will include the formula for the tau coefficient below. Pairs of observations are said ...

View answer
1 answers
0 votes
89 views
Testing for intra-subject agreement across categories
0 votes

No chance-adjusted index of agreement is going to give you a high score: Bennett et al.'s $S = 0.222$ Cohen's $\kappa = 0.267$ Gwet's $\gamma = 0.316$ Krippendorff's $\alpha = 0.211$ Scott's $\pi ...

View answer
1 answers
0 votes
2k views
Calculating interrater agreement for multiple choice questions
1 votes

Kramer (1980) proposed a method for assessing inter-rater reliability for tasks in which raters could select multiple categories for each object of measurement. The intuition behind this method is to ...

View answer
1 answers
1 votes
2k views
Inter-rater reliability measure with multiple categories per item
3 votes

Kramer (1980) proposed a method for assessing inter-rater reliability for tasks in which raters could select multiple categories for each object of measurement. The intuition behind this method is to ...

View answer
1 answers
8 votes
8k views
Quadratic weighted kappa versus linear weighted kappa
8 votes

For four categories, the following linear and quadratic weights would be used. These tables can be read by indexing one rater by row and the other rater by column. For instance, raters would earn 0.33 ...

View answer
1 answers
9 votes
3k views
Computing inter-rater reliability in R with variable number of ratings?
7 votes

The model you referenced in your question is called the "one-way model." It assumes that random row effects are the only systematic source of variance. In the case of inter-rater reliability, rows ...

View answer
1 answers
0 votes
2k views
Intraclass correlation coefficient: zero and negative
1 votes

The intraclass correlation coefficient (ICC) works by partitioning the rating variance into multiple components, e.g., variance associated with items, variance associated with raters, and variance ...

View answer
1 answers
4 votes
2k views
Are there good reasons to use Cohen's Kappa over Gwet's AC1?
Accepted answer
3 votes

I would argue that Cohen's kappa and Gwet's gamma are both problematic approaches to the estimation of inter-rater reliability. Both make a number of assumptions about the behavior of raters that are ...

View answer
1 answers
0 votes
288 views
Computing Inter-Rater Reliability - Different subjects are rated by different subsets of coders
1 votes

The original kappa formula (and that used by SPSS) converts the data into a $q$-by-$q$ confusion matrix (item counts in a category-by-category matrix). This formula required exactly two raters. Later "...

View answer
1 answers
0 votes
74 views
Which reliability index (if any) is appropriate?
1 votes

It depends on what your question is. If you want to know if these sum scores are consistent over time (within each participant), you would need participants to fill out the questions multiple times; ...

View answer
1 answers
1 votes
95 views
Correct approach for computing interrater reliability
1 votes

It's not clear what ev1 and ev2 are in your example or how this constitutes inter-method reliability. I'm also not sure what you are trying to calculate the reliability of. Let's assume you have each ...

View answer
1 answers
1 votes
56 views
What are the most appropriate statistics to calculate using this reliability data?
1 votes

Format the data into a matrix such that each row corresponds to a single test and each column corresponds to a single teacher. You can then calculate the intraclass correlation coefficient (ICC). If ...

View answer
1 answers
0 votes
115 views
Comparison of two measurement methods - determining measurement reliability
1 votes

It's not clear who or what you are quoting, but it seems that they want estimates of what are often referred to as "inter-rater reliability" and "intra-rater reliability." Inter-rater reliability ...

View answer
1 answers
1 votes
224 views
What to report for reliability analysis?
1 votes

You can of course report both intra-rater and inter-rater reliability. Which one is more important in a given study depends on what point you are trying to make and what threat to measurement validity ...

View answer
1 answers
3 votes
2k views
How can I calculate percent agreement and kappa for each category?
Accepted answer
3 votes

The extended formula (for multiple categories and raters) for observed agreement is as follows: $$ A = \frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^{q}\frac{r_{ik}(r_{ik-1})}{r_i(r_i-1)} $$ where $n'$ is the ...

View answer
2 answers
1 votes
173 views
Inter-subject agreement re timings of (unequal number of) events
1 votes

I recommend dividing each time-series into temporal bins. Each bin would take on a value of 1 if the rater pressed the button during that bin and 0 otherwise. You now have a comparable time-series for ...

View answer
1 2 3 4
5