Intraclass correlation in computer tests, The true difference between average and single measures?

Question

I try to produce test/retest coefficient for our computer based executive function test using intraclass correlation coefficient. The test is very structured computer based test and it has hundreds of short Go/Nogo trials. The subjects have executed the test twice few weeks apart each test.

So I have dataframes consisting all single trials (df1) and dataframe (df2) consisting average reaction times per subject ($n=20$) and error percentages per subject.

Question 1: The fixed model is the right model to use, I suppose?

Question 2: What is the true difference between average and single measures at this kind of situations? Which is more recommendable to use, ICC(3,1) or ICC(3,k)?

I suppose using the use of df2 and ICC is best way to evaluate the repeatability of our test.

Answer 1

If you are interested in test-retest reliability, set up your data in a matrix such that each row corresponds to the data from a single subject and each column corresponds to a single test. It sounds like the test was taken twice per subject, so you will have two columns (although more could be accommodated). Thus, if you have $n$ subjects, you will have an $n\times2$ matrix where each cell $x_{ij}$ corresponds to the average "score" (e.g., reaction time or error percentage) of subject $i$ on test $j$. Do not include the scores from individual trials in this analysis.

Question 1: It does sound like the fixed model is appropriate to your measurement situation. However, although the ICC population parameter definitions vary based on whether the column effect is assumed to be random or fixed, the process for calculating the sample estimator is identical in both cases. That is, the estimate will be identical in numerical value for both models and only the interpretation of the estimate will vary (i.e., in how generalizable it is).

Question 2: If you plan to use subjects' scores from a single test to make decisions or for statistical analysis, then use a single-measures ICC. If, however, you plan to test each subject twice and then use their average score across both tests to make decisions or for statistical analysis, then use an average-measures ICC. I assume the former case and thus would recommend ICC(2,1).