What is the right statistical measure for consistency?

Question

I have ordinal data from a Likert survey with $45$ questions and $15$ participants. Every participant took the survey six times, so I have six answers from every participant for every question. All answers are $\in \{1, 2, 3, 4, 5\}$. The participants are not divided into groups.

I want to quantify the consistency with which the participants answered each question. For example, let's say that for some question $q$, we only had two participants $p_1$ and $p_2$ and they answered in the following way:

$p_{1,q}: [2, 2, 2, 2, 2, 2]$, $p_{2,q}: [5, 5, 5, 5, 5, 5]$.

There is zero variability in the six answers each participant gave, so the statistic should have its maximum value.

What would be the appropriate coefficient/statistic to use here?

Answer 1

Intraclass correlation (ICC) is often used to measure rater consistency. In the original article,

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

Shrout & Fleiss consider six different cases. Your situation is case 3 and the resulting ICC index is known as "ICC3":

ICC3: A fixed set of k judges rate each target.

A ready to use implementation of ICC3 can, e.g., be found in the function ICC of the R package psych. For a freely accessible (I think) overview, see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4913118/