0
$\begingroup$

I am trying to calculate reliability between two raters for continuous data. The data is frequency of negative life events for each participant. My data looks like this:

Dataset 1

rater 1: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0

rater 2: 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0

Dataset 2

rater 1: 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0

rater 2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

This is continuous ratio data with a true zero (i.e. 0 indicates that there are no negative life events). The values can go beyond 1 but this is what we ended up with. My problem is I'm getting a negative ICC for dataset 1 (-0.205) and zero for dataset 2. How do I interpret this and report it in the write up?

Improve this question
$\endgroup$
7
  • $\begingroup$ Welcome to CV. First of all, this doesn't look like "continuous ratio data" to me. Rather, it looks like ordinal, zero heavy count data. Next, based on these two datasets, there are no points of agreement in #1 and rater 2 in #2 has no life events at all. Finally, what is being rated? Is it a time series? $\endgroup$
    – user78229
    Commented Apr 1, 2016 at 11:12
  • $\begingroup$ Sorry, I should have clarified what was being rated. Each point of data = number of negative life events per participant. You're right, it is count data. Raters decided if each event was negative or not and the values are how many were judged negative for each participant. It just ended up that many events were not negative, hence the zero heavy counts. $\endgroup$
    – StatsDummy
    Commented Apr 1, 2016 at 11:22
  • $\begingroup$ So, there are 16 participants being rated by two people for each dataset? How do the datasets differ? Are there 4 different raters? $\endgroup$
    – user78229
    Commented Apr 1, 2016 at 11:25
  • $\begingroup$ I tried to simplify in presenting the data here and maybe have overdone it! They are actually from the same dataset (and same participants). Dataset 1 is for life events that are long term and dataset 2 is for life events that are short term. They are mutually exclusive in our data. $\endgroup$
    – StatsDummy
    Commented Apr 1, 2016 at 11:29
  • $\begingroup$ Forgot to add, its the same two raters in each dataset. $\endgroup$
    – StatsDummy
    Commented Apr 1, 2016 at 11:30

1 Answer 1

Reset to default
2
$\begingroup$

The intraclass correlation coefficient (ICC) works by partitioning the rating variance into multiple components, e.g., variance associated with items, variance associated with raters, and variance associated with measurement error. The ICC value is set to equal the fraction of "relevant" variance that is associated with items. Using this approach, when the total rating variance is low, it is almost impossible to achieve a high ICC value. In your example, there is very little variance and thus the ICC values are low despite the raters agreeing on their ratings of 0 for most items.

In this case, due to the low variance, an alternative approach is probably needed. Rather than using the ICC, you might consider using a categorical agreement coefficient with a non-nominal weighting scheme. There are many possible options, including a weighted kappa coefficient or S score.

Click here to view more information and access functions for calculating these coefficients.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.