# Inter-rater reliability for "time-series"

I would like to calculate the inter-rater reliability between 3 raters for some kind of time-series. Actually, I'm not quite sure, if I can call my data time-series but I will give you an example, so hopefully, you will get my idea then.

I do some ratings of the behavior of persons in a video. There are 4 categories (can be treated as ordinal data) and there will be a rating for each time frame. So, the data will somehow look like this.

Person A

Time frame Rater 1 Rater 2 Rater 3
1 2 2 2
2 2 2 3
3 2 2 3
4 3 3 3
5 3 3 3
6 3 3 1
7 3 3 3
8 2 2 2
9 2 2 1
10 2 2 2

And I have this kind of data for every person (approx. 30 persons). Now, I would like to calculate, if the raters agree on their ratings over all persons. I tried to google it, but I just found inter rater reliability indices in cases of one rating for each person and not for these multiple time points.

Maybe somebody has an idea.

• How about computing your index of agreement for each target and then average the 30 values you get in some principled way? Mar 3, 2021 at 13:33
• Yeah, I also thought about doing something like this, but I wasn't sure if this could work. In this case, I thought about something like ICC or Fleiss' Kappa Mar 3, 2021 at 13:38

You have ordinal categories and multiple raters. A generalized coefficient of agreement or chance-adjusted agreement can handle this no problem. This approach is described in Gwet (2014). For example, the generalization of Fleiss' kappa (and Scott's pi) is provided below:

$$r_{ik}^\star = \sum_{l=1}^q w_{kl}r_{il}$$

$$p_o = \frac{1}{n'} \sum_{i=1}^{n'} \sum_{k=1}^q \frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$

$$\pi_k = \frac{1}{n}\sum_{i=1}^n \frac{r_{ik}}{r_i}$$

$$p_c = \sum_{k,l}^q w_{kl} \pi_k \pi_l$$

$$\kappa = \frac{p_o-p_c}{1 - p_c}$$

where

$$q$$ is the total number of categories

$$w_{kl}$$ is the weight associated with two raters assigning an item to categories $$k$$ and $$l$$

$$r_{il}$$ is the number of raters that assigned item $$i$$ to category $$l$$

$$n'$$ is the number of items coded by two or more raters

$$r_{ik}$$ is the number of raters that assigned item $$i$$ to category $$k$$

$$r_i$$ is the number of raters that assigned item $$i$$ to any category

$$n$$ is the total number of items

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.