Disagreement between ICC and paired t-test

Question

I am trying to determine whether the values obtained by an algorithm calculation are systematically different from the the criterion values.

The criterion values are obtained via manual observation with 3 different judges. The values obtained are as follows:

judge1_count = [1456 1430 1471 3024 3802 4334 3812 4140 1089 1860 2201 1107 1134];
judge2_count = [1458 1427 1473 3023 3835 4350 3791 4129 1090 1867 2212 1115 1147];
judge3_count = [1452 1441 1473 3030 3845 4360 3820 4160 1074 1863 2216 1119 1139];

Each column represents a different independent sample from which the 3 judges give a count

I first test for inter-rater reliability by calculating ICC(2,1) (two-way random effect, single model), giving me an ICC value of about 0.99, which makes sense just by observing how close the counts are for each sample.

Next I introduce the algorithm counts:

algo_count = [1434 1422 1453 2973 3688 4061 3703 4072 1018  1837 2180 1090 1116]

And once again apply ICC(2,1), giving me an ICC of again around 0.99. However, when I apply a paired t-test of the algorithm count and the criterion count (where the criterion count is averaged across the 3 judges per sample), the null hypothesis is rejected (p = 0.0078), and by observation it can be seen that the algorithm counts are systematically smaller than the criterion counts, so this does make some sense.

My question then is that are these results logical, or have I missed something in regard to how I apply the ICCs? This is my first time using them so I'm hesitant about whether I've used this function properly. For reference I repeated the task in both MATLAB and R and got out the same ICC values.

Answer 1

Thanks for the clarification in the comments. I will try to elaborate an argument for the differences.

Let's start with the paired t-tests statistics for no difference in two samples is defined as

$t=\mu_d/s_d\sqrt(n)$

where, $\mu_d$ is the mean of the difference, $s_d$ is the standard deviation of the difference and $n$ is the number of observations. Thus under the assumptions that the difference is normal distributed we get the statistic for the mean difference.

The ICC(2,1) is based on a linear mixed model.

$y_{ij} = \alpha + b_i + u_{j} + \epsilon_{ij}$

where $y_{ij}$ is the the observed outcome for subject $i$ by judge $j$. The coefficient $\alpha$ is the intercept, $b_i$ is the between-cluster variance for subjects, $u_j$ is the between-cluster variance for judges and $\epsilon_{ij}$ is the additional variation, not accounted for by judges or individuals. Thus the interclass correlation is based on a linear mixed model, where the outcome is expected to be approximately normal distributed after accounting for cluster variation (since the error terms are usually assumed normal distributed actually the outcome is assumed to be normal distributed).

The interclass correlation is now the proportion of variation explained by subject differences. That is

$ICC(2,1)= b_i/(u_j+b_i+\epsilon_{ij})$

Thus if most of the variation is due to subject differences, we get a high ICC.

To relate to the t-test. In the paired t-test we only consider the overall variation, similar to the model

$d_i = \alpha + \epsilon_{i}$

Thus the variation due to judge and individual will be considered as unexplained variation. That means we can only see a difference in means, not from which the variation comes. In the ICC we see that most of the variation is due to individual differences in the measured subjects, and not due to differences in the judges measurements.

Thus we have to completely different variance structures, and unless the individual and judge variation are neglectable small, we would not expect to get comparable results.

I hope this clarifies your problem.