I would like to calculate the inter-rater reliability between 3 raters for some kind of time-series. Actually, I'm not quite sure, if I can call my data time-series but I will give you an example, so hopefully, you will get my idea then.

I do some ratings of the behavior of persons in a video. There are 4 categories (can be treated as ordinal data) and there will be a rating for each time frame. So, the data will somehow look like this.

Person A

Time frame Rater 1 Rater 2 Rater 3
1 2 2 2
2 2 2 3
3 2 2 3
4 3 3 3
5 3 3 3
6 3 3 1
7 3 3 3
8 2 2 2
9 2 2 1
10 2 2 2

And I have this kind of data for every person (approx. 30 persons). Now, I would like to calculate, if the raters agree on their ratings over all persons. I tried to google it, but I just found inter rater reliability indices in cases of one rating for each person and not for these multiple time points.

Maybe somebody has an idea.

Thank you in advance :)

  • $\begingroup$ How about computing your index of agreement for each target and then average the 30 values you get in some principled way? $\endgroup$
    – mdewey
    Mar 3 at 13:33
  • $\begingroup$ Yeah, I also thought about doing something like this, but I wasn't sure if this could work. In this case, I thought about something like ICC or Fleiss' Kappa $\endgroup$
    – HelliLu
    Mar 3 at 13:38

You have ordinal categories and multiple raters. A generalized coefficient of agreement or chance-adjusted agreement can handle this no problem. This approach is described in Gwet (2014). For example, the generalization of Fleiss' kappa (and Scott's pi) is provided below:

$$ r_{ik}^\star = \sum_{l=1}^q w_{kl}r_{il} $$

$$ p_o = \frac{1}{n'} \sum_{i=1}^{n'} \sum_{k=1}^q \frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)} $$

$$ \pi_k = \frac{1}{n}\sum_{i=1}^n \frac{r_{ik}}{r_i} $$

$$ p_c = \sum_{k,l}^q w_{kl} \pi_k \pi_l $$

$$ \kappa = \frac{p_o-p_c}{1 - p_c} $$


$q$ is the total number of categories

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$

$r_{il}$ is the number of raters that assigned item $i$ to category $l$

$n'$ is the number of items coded by two or more raters

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

$n$ is the total number of items

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.