# Testing two raters' time-point data for interrater reliability [closed]

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of observations). I have two questions:

1) What do I call these data? "Time-series data" seems too general and usually refers to metric data changing continuously over time (while I have just points of data along the time line). Under "time-point data" I don't find problems of the kinds described in question (2).

2) What indices for interrater reliability can I use (in R)? (If an index requires defining how much offset is tolerated, that could be 0.120 seconds.)

example data (in seconds)

rater1:

181.23
181.566
181.986
182.784
183.204
191.352
193.956
195.426
197.568
197.82
198.576
202.02
205.8
206.136
208.53
209.034
216.216
220.08
220.584
230.706
238.266
238.518
239.442
241.5
241.836
244.398


rater2:

181.902
182.784
183.204
193.956
195.384
197.694
197.82
198.576
199.5
202.146
205.8
206.136
208.53
216.258
219.576
220.542
222.096
222.558
226.002
228.312
229.11
230.244
230.496
230.832
231.504
232.554
238.266
238.518
238.602
238.938
241.5
241.836
244.272


I asked about a different part of this problem already here. But I guess that was not the ideal community and I supplied insufficient data.

• Some common interrater reliability measures for continuous data are the intraclass correlation coefficient and Lin's Concordance Coefficient. I am unaware of a concordance measure specifically for time series though. Are the readings paired at all? My first impression is they aren't, since one list has 26 values and the other has 33. Is there a standard of truth, for example you know when the phenomenon was generated? – Ashe Apr 15 '15 at 13:51
• Also, this link might be of use. stats.stackexchange.com/questions/5696/… – Ashe Apr 15 '15 at 13:56
• @Joshua: Since high ICC values indicate significant differences across groups, a high ICC would also indicate that the raters are not at all in sync. I'm not sure how much this index work the other way around. Can a low value be interpreted to indicate high interrater reliability? – pointingeye Apr 17 '15 at 10:45
• Be wary of conflating correlation with significance. A high ICC (close to 1.0) would indicate strong interrater reliability (the two raters return nearly similar results). The p-value, which is a separate measure, would indicate how likely the ICC value is compared to some null hypothesis (usually $r_0$=0). The irr package in R has an icc function that will give both values. Note though that ICC requires paired measurements, meaning the list of 26 and 33 are not strictly paired. One rater found phenomena that the other missed. – Ashe Apr 17 '15 at 13:51

1)

When we want to estimate rates we really want to estimate parameters for the distribution or process leading to failure. This in itself is meant for predicting time to failure/event.

The times between- or to events of interest are usually called

• event-times
• survival-times
• life lengths
• failure-times
• hitting-time
• stopping-time
• waiting-time

Data where we have time-series for a failure process measured in wall-clock time or subjective time (i.e timestep is one relapse, one cycle, one random measure point etc) is more broadly called data with time varying covariates. I like to call this type of data 'event data' as all we need is the binary events $$v_t^n\in \{0,1\}$$ for each sequence $$n$$ at each timestep $$t$$ and some aggregated features $$x_t^n$$. This can then be transformed to answer any of the questions of rates etc.

Naming differs from community to community. It's horrible. Some groups dealing with this type of data are

• People doing Run-to-failure-experiments. PHM. This gives a great overview of ML-perspectives and some data
• Survival Analysis-community. Beware of 1000-page books consisting of 99% medical jargon.
• Stochastic process/SDE-community. Here you usually want to model a stochastic process reaching a threshold and infer the rate or assume the rate itself is the stochastic process. Awesome 2 page article intro
• Churn community, i.e "BI-experts" typically predicting whether event happens in $$\tau$$ steps at each point $$t$$. See 2 which briefly discusses this in another context

2) Concordance index is a fancy word for AUC which is a fancy word for GINI which is a fancy normalized Mann-Whitney U-statistic. If you mash up your two datasets and add a binary column indicating sequence 1 or sequence 2 you have the same data as you'd have if a ML-algo outputted a (non-normalized) score for binary prediction. Proceed as if you're evaluating this model with sequence-indicator column the ground truth.

I don't think you should be interpreting this as "time series" data as you really aren't interested in analyzing the data per se, but the agreement among the coders. It is, as you have indicated, an interrater reliability problem. As @Joshua pointed out, you could probably use ICC. I'd also recommend you look into use of Cohen's or Fleiss' kappa, or Kendall's tau. You might also look into using Krippendorff’s alpha. Friedman's Chi-square test might also be useful, depending on the type of data you've collected.