3
$\begingroup$

My data consists of two raters interpreting one specific phenomenon to occur at different points in time (the observations are not paired, the raters actually identified different amounts of observations). I have two questions:

1) What do I call these data? "Time-series data" seems too general and usually refers to metric data changing continuously over time (while I have just points of data along the time line). Under "time-point data" I don't find problems of the kinds described in question (2).

2) What indices for interrater reliability can I use (in R)? (If an index requires defining how much offset is tolerated, that could be 0.120 seconds.)

example data (in seconds)

rater1:

181.23  
181.566 
181.986 
182.784 
183.204 
191.352 
193.956 
195.426 
197.568 
197.82  
198.576 
202.02  
205.8   
206.136 
208.53  
209.034 
216.216 
220.08  
220.584 
230.706 
238.266 
238.518 
239.442 
241.5   
241.836 
244.398

rater2:

181.902 
182.784 
183.204 
193.956 
195.384 
197.694 
197.82  
198.576 
199.5   
202.146 
205.8   
206.136 
208.53  
216.258 
219.576 
220.542 
222.096 
222.558 
226.002 
228.312 
229.11  
230.244 
230.496 
230.832 
231.504 
232.554 
238.266 
238.518 
238.602 
238.938 
241.5   
241.836 
244.272

I asked about a different part of this problem already here. But I guess that was not the ideal community and I supplied insufficient data.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Some common interrater reliability measures for continuous data are the intraclass correlation coefficient and Lin's Concordance Coefficient. I am unaware of a concordance measure specifically for time series though. Are the readings paired at all? My first impression is they aren't, since one list has 26 values and the other has 33. Is there a standard of truth, for example you know when the phenomenon was generated? $\endgroup$
    – Ashe
    Commented Apr 15, 2015 at 13:51
  • $\begingroup$ Also, this link might be of use. stats.stackexchange.com/questions/5696/… $\endgroup$
    – Ashe
    Commented Apr 15, 2015 at 13:56
  • $\begingroup$ @Joshua: Since high ICC values indicate significant differences across groups, a high ICC would also indicate that the raters are not at all in sync. I'm not sure how much this index work the other way around. Can a low value be interpreted to indicate high interrater reliability? $\endgroup$
    – pointingeye
    Commented Apr 17, 2015 at 10:45
  • $\begingroup$ Be wary of conflating correlation with significance. A high ICC (close to 1.0) would indicate strong interrater reliability (the two raters return nearly similar results). The p-value, which is a separate measure, would indicate how likely the ICC value is compared to some null hypothesis (usually $r_0$=0). The irr package in R has an icc function that will give both values. Note though that ICC requires paired measurements, meaning the list of 26 and 33 are not strictly paired. One rater found phenomena that the other missed. $\endgroup$
    – Ashe
    Commented Apr 17, 2015 at 13:51

2 Answers 2

Reset to default
1
$\begingroup$

1)

I'm not sure about your question about your data so I'll try to give you the tools to start googling because I've been in your situation too.

When we want to estimate rates we really want to estimate parameters for the distribution or process leading to failure. This in itself is meant for predicting time to failure/event.

The times between- or to events of interest are usually called

  • event-times
  • survival-times
  • life lengths
  • failure-times
  • hitting-time
  • stopping-time
  • waiting-time

Data where we have time-series for a failure process measured in wall-clock time or subjective time (i.e timestep is one relapse, one cycle, one random measure point etc) is more broadly called data with time varying covariates. I like to call this type of data 'event data' as all we need is the binary events $v_t^n\in \{0,1\}$ for each sequence $n$ at each timestep $t$ and some aggregated features $x_t^n$. This can then be transformed to answer any of the questions of rates etc.

Naming differs from community to community. It's horrible. Some groups dealing with this type of data are

  • People doing Run-to-failure-experiments. PHM. This gives a great overview of ML-perspectives and some data
  • Survival Analysis-community. Beware of 1000-page books consisting of 99% medical jargon.
  • Stochastic process/SDE-community. Here you usually want to model a stochastic process reaching a threshold and infer the rate or assume the rate itself is the stochastic process. Awesome 2 page article intro
  • Churn community, i.e "BI-experts" typically predicting whether event happens in $\tau$ steps at each point $t$. See 2 which briefly discusses this in another context

2) Concordance index is a fancy word for AUC which is a fancy word for GINI which is a fancy normalized Mann-Whitney U-statistic. If you mash up your two datasets and add a binary column indicating sequence 1 or sequence 2 you have the same data as you'd have if a ML-algo outputted a (non-normalized) score for binary prediction. Proceed as if you're evaluating this model with sequence-indicator column the ground truth.

Improve this answer
$\endgroup$
0
$\begingroup$

I don't think you should be interpreting this as "time series" data as you really aren't interested in analyzing the data per se, but the agreement among the coders. It is, as you have indicated, an interrater reliability problem. As @Joshua pointed out, you could probably use ICC. I'd also recommend you look into use of Cohen's or Fleiss' kappa, or Kendall's tau. You might also look into using Krippendorff’s alpha. Friedman's Chi-square test might also be useful, depending on the type of data you've collected.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.