I am studying the reproducibility of different methods using the Intra-Class Correlation (ICC) and the Coeficient of Variation (CV). Some methods are presenting an excellent ICC (~.90) and a poor (for what I would like to do with it) CV (~20%). How can I interpret this result?

The methods are extracting clinicals biomarkers, and the methods are applied two time on the same sample in different conditions (observers, time, acquisition). The goal is to observe wether the method is reproducible. The number of samples is quite small (~20).

In many studies, only ICC is used but there is some problemes for the comparaison between studies because it is too dependants of the samples repartition. So I wanted to have other evaluation parameters but now I find myself with divergent opinion on the reproducibility of the method. And I don't know how to combine the two parameters to have a solid interpretation ( inconclusive for identified reasons is a solid one).

OF NOTE: CV is from duplicates values but calculated as in the litterature of my specific field as the standard deviation of the differences over the global mean. This formula depends on the bias and is not centered on the mean, which is bad, I think. (For the correct formula, see MedCalc.)

[I don't have the outputs on hand right now]

  • 3
    $\begingroup$ Can you provide more information about your study, your data, your analyses & your goals? Can you paste in your output? I'm not sure if this can be answered at present. $\endgroup$ – gung - Reinstate Monica Sep 23 '16 at 16:57

In answering this, I am taking the perspective of a research psychologist or neuroscientist, who is measuring multiple persons repeatedly in some outcome (e.g., some score on an intelligence test or a property of a given brain region). He or she is primarily interested in assessing how well the outcome may function as a marker for some distal outcome.

Intra-class correlation (ICC) and coefficient of variation (CV) both are standardized measures of the within-person variability in a given measure. The within-person variability as a measure of precision (e.g., expressed as standard deviation) is expressed in some unit and, often, it makes sense to derive a unit-free, standardized measure for comparability across studies. ICC and CV achieve this in a different way:

CV is the ratio of within-person standard deviation and mean.

ICC is the ratio of the true-score variance to total observed variance. We can rewrite ICC as a squashed version of the ratio of between-person variance and within-person variance. Let $s(x)=\frac{1}{1+\frac{1}{x}}$ be a squashing function, let $\sigma^2_B$ be the between-person variance and $\sigma^2_W$ the within-person variance, then $s(\frac{\sigma^2_B}{\sigma^2_W})=\frac{\sigma^2_B}{\sigma^2_B+\sigma^2_W}=ICC$

In the above sense, both measures are essentially signal-to-noise ratios (respectively noise-to-signal ratios) but "signal" is defined differently in each.

Again, the principled difference is the value against which we standardize our estimates of within-person variability.

ICC and CV must not agree, at all. It all depends what kind of precision/reliability you want to convey. CV, which relates precision to the sample mean, expresses how far in within-person-sd-units we are away from zero and, thus, is a measure of how well we can detect something away from zero. ICC, in contrast, relates to the between-person variance. Thus, it says something how well we can discriminate between individuals. As a psychologist, one is typically highly interested in measures that can discriminate between persons, such that the measure can have diagnostic value.

When comparing both, it makes sense to compare ICC and 1/CV. In this metric, for both measures, higher means more precise and lower means less precise.

In the following fake data, five persons were each measured 10 times on variable X.

enter image description here

The sample mean is about 0, so 1/CV is very low. So, it's rather difficult to detect values to be away from zero. At the same time, ICC is rather high since the within-variability is much smaller than the between-person variability and the measure is highly discriminating between persons (though, we may have difficulties in assessing whether it is truly different from zero for each person).

The other extreme would be the following dataset:

enter image description here

Here, ICC is pretty low as within-person and between-person variability are identical. The measure does not discriminate well between persons. At the same time, 1/CV would be rather high, since the mean is large and so will be the mean to SD ratio.

| cite | improve this answer | |
  • $\begingroup$ Here is our open-access article discussing these issues in the context of human cognitive neuroscience: elifesciences.org/articles/35718 $\endgroup$ – Brandmaier Jul 3 '18 at 7:05
  • $\begingroup$ Perfect article, it is answering a lot more questions I had. Thank you. $\endgroup$ – Jimolrame Sep 21 '18 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.