Inter-rater reliability for "time-series" I would like to calculate the inter-rater reliability between 3 raters for some kind of time-series. Actually, I'm not quite sure, if I can call my data time-series but I will give you an example, so hopefully, you will get my idea then.
I do some ratings of the behavior of persons in a video. There are 4 categories (can be treated as ordinal data) and there will be a rating for each time frame. So, the data will somehow look like this.
Person A




Time frame
Rater 1
Rater 2
Rater 3




1
2
2
2


2
2
2
3


3
2
2
3


4
3
3
3


5
3
3
3


6
3
3
1


7
3
3
3


8
2
2
2


9
2
2
1


10
2
2
2




And I have this kind of data for every person (approx. 30 persons). Now, I would like to calculate, if the raters agree on their ratings over all persons. I tried to google it, but I just found inter rater reliability indices in cases of one rating for each person and not for these multiple time points.
Maybe somebody has an idea.
Thank you in advance :)
 A: You have ordinal categories and multiple raters. A generalized coefficient of agreement or chance-adjusted agreement can handle this no problem. This approach is described in Gwet (2014). For example, the generalization of Fleiss' kappa (and Scott's pi) is provided below:
$$
r_{ik}^\star = \sum_{l=1}^q w_{kl}r_{il}
$$
$$
p_o = \frac{1}{n'} \sum_{i=1}^{n'} \sum_{k=1}^q \frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}
$$
$$
\pi_k = \frac{1}{n}\sum_{i=1}^n \frac{r_{ik}}{r_i}
$$
$$
p_c = \sum_{k,l}^q w_{kl} \pi_k \pi_l
$$
$$
\kappa = \frac{p_o-p_c}{1 - p_c}
$$
where
$q$ is the total number of categories
$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$
$r_{il}$ is the number of raters that assigned item $i$ to category $l$
$n'$ is the number of items coded by two or more raters
$r_{ik}$ is the number of raters that assigned item $i$ to category $k$
$r_i$ is the number of raters that assigned item $i$ to any category
$n$ is the total number of items

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics.
