Inter-rater reliability for "time-series"

Question

I would like to calculate the inter-rater reliability between 3 raters for some kind of time-series. Actually, I'm not quite sure, if I can call my data time-series but I will give you an example, so hopefully, you will get my idea then.

I do some ratings of the behavior of persons in a video. There are 4 categories (can be treated as ordinal data) and there will be a rating for each time frame. So, the data will somehow look like this.

Person A

Time frame	Rater 1	Rater 2	Rater 3
1	2	2	2
2	2	2	3
3	2	2	3
4	3	3	3
5	3	3	3
6	3	3	1
7	3	3	3
8	2	2	2
9	2	2	1
10	2	2	2

And I have this kind of data for every person (approx. 30 persons). Now, I would like to calculate, if the raters agree on their ratings over all persons. I tried to google it, but I just found inter rater reliability indices in cases of one rating for each person and not for these multiple time points.

Maybe somebody has an idea.

Thank you in advance :)

Answer 1

You have ordinal categories and multiple raters. A generalized coefficient of agreement or chance-adjusted agreement can handle this no problem. This approach is described in Gwet (2014). For example, the generalization of Fleiss' kappa (and Scott's pi) is provided below:

$$ r_{ik}^\star = \sum_{l=1}^q w_{kl}r_{il} $$

$$ p_o = \frac{1}{n'} \sum_{i=1}^{n'} \sum_{k=1}^q \frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)} $$

$$ \pi_k = \frac{1}{n}\sum_{i=1}^n \frac{r_{ik}}{r_i} $$

$$ p_c = \sum_{k,l}^q w_{kl} \pi_k \pi_l $$

$$ \kappa = \frac{p_o-p_c}{1 - p_c} $$

where

$q$ is the total number of categories

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$

$r_{il}$ is the number of raters that assigned item $i$ to category $l$

$n'$ is the number of items coded by two or more raters

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

$n$ is the total number of items

---

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.