10
$\begingroup$

I have calculated the ICC values for two groups and would now like to compare the ICC values to determine if the groups differ in their repeatability. In the literature people have simply used t-tests to compare repeatability but it is unclear to me how to do this.

For example, with the dummy data:

ID  gr  day behaviour
1   1   1   0.361
2   1   1   0.232
3   1   1   0.240
4   1   1   0.693
5   1   1   0.483
6   1   1   0.267
7   2   1   0.180
8   2   1   0.515
9   2   1   0.485
10  2   1   0.567
11  2   1   0.000
12  2   1   0.324
1   1   2   0.055
2   1   2   0.407
3   1   2   0.422
4   1   2   0.174
5   1   2   0.613
6   1   2   0.311
7   2   2   0.631
8   2   2   0.283
9   2   2   0.512
10  2   2   0.127
11  2   2   0.000
12  2   2   0.000

I can get the repeatability measures for group 1 and 2 as follows:

library(ICC)
g1 <- ICCest(ID, behaviour, data=dummy[dummy$gr=="1",])
g2 <- ICCest(ID, behaviour, data=dummy[dummy$gr=="2",])

But how can I now determine if the repeatability of group1 is different from group2?

Improve this question
$\endgroup$
4
  • $\begingroup$ I notice that each of your groups has only 2 clusters. This is definitely not ideal, and is surely the reason why the confidence intervals around both of your ICC estimates are extremely wide (which I see from the answer of @JamesStanley below), at least for this sample dataset you've provided. In your actual dataset, do you have only 2 clusters per group, or (hopefully) more clusters than this? If more, how many per group? $\endgroup$
    – Jake Westfall
    Commented Jan 10, 2014 at 3:34
  • $\begingroup$ What do you mean exactly with 2 clusters? I tested two groups twice yes, I don't see why that is not ideal? I now ran permutation tests to compare the ICC of both groups (r = 0.77, 95% CI: 0.54, 0.91 and for group 2 r = 0.24, 95% CI: 0.07, 0.57) which reveals group 1 has significantly higher repeatability that group 2. $\endgroup$
    – crazjo
    Commented Jan 10, 2014 at 8:28
  • 1
    $\begingroup$ Maybe we should back up. The ICC is applicable to clustered (i.e., multilevel) data. It is a way of measuring how similar on average are two observations drawn from the same cluster, relative to two observations drawn at random from the dataset ignoring clustering. In practice it is computed as the ratio of between-cluster variance to total variance. So if you only have 2 clusters in each of your groups, then the estimate of between-cluster variance is only based on 2 data points. Imagine attempting to estimate, say, the mean or standard deviation of a dataset consisting of only 2 data points! $\endgroup$
    – Jake Westfall
    Commented Jan 10, 2014 at 17:28
  • $\begingroup$ As for your permutation test, I'd be very interested to see exactly how this was conducted. I was thinking of posting a solution based on a bootstrap or permutation test. Note that this kind of thing must be done pretty carefully with multilevel data! $\endgroup$
    – Jake Westfall
    Commented Jan 10, 2014 at 17:32

1 Answer 1

Reset to default
1
$\begingroup$

Leaving aside material issues about the study question and demo data (and sample size for getting a reasonable estimate of an ICC), the output you are getting from the ICCest function has confidence intervals attached: as a starting point for comparing groups, you could consider whether there is overlap between each confidence interval and the other group's point estimate of the ICC.

At any rate, reporting the point estimate of the ICC and the confidence interval for each group is going to be more useful (and hence I'd recommend reporting these in any instance) than reporting just the point estimates and the result of some kind of hypothesis test.

dummy <- structure(list(ID = c(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L,
                  11L, 12L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L, 11L, 12L), 
           gr = c(1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
                  1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 2L, 2L), 
           day = c(1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 
                   2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), 
           behaviour = c(0.361, 0.232, 0.24, 0.693, 0.483, 0.267, 0.18, 0.515, 0.485,
                         0.567, 0, 0.324, 0.055, 0.407, 0.422, 0.174, 0.613, 0.311, 
                         0.631, 0.283, 0.512, 0.127, 0, 0)), 
           .Names = c("ID", "gr", "day", "behaviour"), 
          class = "data.frame", row.names = c(NA, -24L))

library(ICC)
ICCest(ID, behaviour, data=dummy[dummy$gr=="1",])
# First few lines of console output:
#$ICC
#[1] -0.1317788
#$LowerCI
#[1] -0.7728603
#$UpperCI
#[1] 0.6851783

ICCest(ID, behaviour, data=dummy[dummy$gr=="2",])
# First few lines of console output:
#$ICC
#[1] 0.1934523
#$LowerCI
#[1] -0.6036826
#$UpperCI
#[1] 0.8233986
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks James, the output I got from the actual data showed for group 1: r = 0.77, 95% CI: 0.54, 0.91 and for group 2 r = 0.77, 95% CI: 0.07, 0.57. You can see group 1 had relatively very high consistency while group 2 on average moderate consistency but the confidence intervals from the two groups barely overlap. Could I write in a manuscript group 2 was considerably less consistent than group 1 (above stats)? I understand you suggest this would be better than running a test to compare them but still it would be nice to know how to compare the ICC values of two groups. $\endgroup$
    – crazjo
    Commented Dec 11, 2013 at 8:48
  • $\begingroup$ I'd feel comfortable reading assessing those two ICCs as printed in your comment as indicating differen reliabilities (I'm assuming the group 2 r has a typo and is in fact ~=0.32) -- the overlap is between the confidence interval ends, rather than a given confidence interval and the opposite point estimate, which if these were differences in means would probably correspond to a "significant" t-test result at p < 0.05. $\endgroup$
    – James Stanley
    Commented Dec 13, 2013 at 4:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.