I have a dataset with more than 100 repeated measures of observations across multiple subjects. I want to estimate the agreement between two of the variables measured repeatedly. Note that each observation is categorical with 4 levels. I have two options:
- I can calculate Cohen's kappa for the entire data set
- I can just get a confusion matrix from the dataset
However, in both these cases, it is assumed that the observations are independent. However, my dataset being of repeated measures kind, I am sure there is going to be some dependence among the observations. Can anyone suggest the right way to account for this dependence by still estimating some kind of intraclass-agreement?