2
$\begingroup$

It's been known since Spearman (1904) that one can adjust an observed Pearson correlation for measurement error using this formula (R code with tidyverse attached):

true_r = observed_r / sqrt(rel_x * rel_y)

Thus, for instance, if we have an observed correlation of .4, and reliabilities of x and y of .6 and .8, we get:

true_r = 0.577 = .4 / sqrt(.6 * .8)

My question is: how does one do the equivalent correction for an intraclass correlation? I know, people normally use ICC as a measure of reliability, but I need to use it as a metric of within group similarity across raters.

Here things get complicated because there are multiple types of ICC, as well as single and average versions (following the taxonomy in psych package, at least). In my case, I am interested in the single rater case, type irrelevant (values are identical for my purposes).

Suppose we have 2 targets rated each 10 times. Suppose their true scores are 0 and -1, respectively. We can easily simulate some data for this and calculate the ICC values:

set.seed(1)
n = 10
testdf = data_frame(
  a = rnorm(n),
  b = rnorm(n, mean = -1)
)
testdf
# A tibble: 10 x 2
        a       b
    <dbl>   <dbl>
 1 -0.626  0.512 
 2  0.184 -0.610 
 3 -0.836 -1.62  
 4  1.60  -3.21  
 5  0.330  0.125 
 6 -0.820 -1.04  
 7  0.487 -1.02  
 8  0.738 -0.0562
 9  0.576 -0.179 
10 -0.305 -0.406 
psych::ICC(testdf %>% t())
Call: psych::ICC(x = testdf %>% t())

Intraclass correlation coefficients 
                         type  ICC   F df1 df2     p lower bound upper bound
Single_raters_absolute   ICC1 0.26 4.5   1  18 0.049      -0.026           1
Single_random_raters     ICC2 0.26 4.5   1   9 0.064      -0.027           1
Single_fixed_raters      ICC3 0.26 4.5   1   9 0.064      -0.040           1
Average_raters_absolute ICC1k 0.78 4.5   1  18 0.049      -0.343           1
Average_random_raters   ICC2k 0.78 4.5   1   9 0.064      -0.355           1
Average_fixed_raters    ICC3k 0.78 4.5   1   9 0.064      -0.620           1

 Number of subjects = 2     Number of Judges =  10

So, for single raters we get .26. The expected value is actually .33, which one sees if one uses a larger sample size:

n2 = 10e3

testdf2 = data_frame(
  a = rnorm(n2),
  b = rnorm(n2, mean = -1)
)

psych::ICC(testdf2 %>% t())
Call: psych::ICC(x = testdf2 %>% t())

Intraclass correlation coefficients 
                         type  ICC    F df1   df2 p lower bound upper bound
Single_raters_absolute   ICC1 0.33 4946   1 19998 0        0.09           1
Single_random_raters     ICC2 0.33 4970   1  9999 0        0.09           1
Single_fixed_raters      ICC3 0.33 4970   1  9999 0        0.09           1
Average_raters_absolute ICC1k 1.00 4946   1 19998 0        1.00           1
Average_random_raters   ICC2k 1.00 4970   1  9999 0        1.00           1
Average_fixed_raters    ICC3k 1.00 4970   1  9999 0        1.00           1

 Number of subjects = 2     Number of Judges =  10000

So far, so good. Now, imagine there is some measurement error in the estimates above. In fact, suppose we add some random measurement error corresponding to reliabilities from .9 to .1. Then we get this:

#reduce reliability while preserving distribution
reduce_reliability = function(x, reliability) {
  x * sqrt(reliability) + rnorm(length(x), mean = 0, sd = (sqrt(1 - reliability)))
}

#loop and get values
rels = seq(.9, .1, by = -.1)
names(rels) = rels
ICC_noise = map_dbl(rels, function(r) {
  #copy original
  tmp = testdf2

  #add error
  tmp$a = tmp$a %>% reduce_reliability(reliability = r)
  tmp$b = tmp$b %>% reduce_reliability(reliability = r)

  #ICC
  psych::ICC(tmp %>% t()) %>% .$results %>% .[[1, 2]]
})
ICC_noise

   0.9    0.8    0.7    0.6    0.5    0.4    0.3    0.2    0.1 
0.3048 0.2845 0.2602 0.2187 0.2047 0.1688 0.1357 0.0902 0.0420 

So, we see an expected decline in the values with increasing measurement error, but how does one adjust this back so we always get .33 here? Dividing by sqrt(reliability) as with the Pearson correlation gives incorrect results:

ICC_noise / sqrt(rels)
  0.9   0.8   0.7   0.6   0.5   0.4   0.3   0.2   0.1 
0.322 0.316 0.317 0.292 0.279 0.271 0.242 0.211 0.166 

ICC_noise / rels
  0.9   0.8   0.7   0.6   0.5   0.4   0.3   0.2   0.1 
0.339 0.353 0.379 0.377 0.395 0.428 0.441 0.472 0.525 

So, appears one has to use a different formula. However, I have been unable to find one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.