1
$\begingroup$

I have two datasets in which I probably need to use the intraclass correlation coefficient, and I want to make sure I choose the right type.

Dataset 1: I had 20 randomly selected human subjects rate 50 faces based on their attractiveness on an ordinal scale from 1-5. Each subject rated the same faces. I want to know how similar their ratings are to each other. Which method should I use? 1-k?

Dataset 2: I have 20 4x4 confusion matrices, one for each human subject. I've been asked by a reviewer to measure how similar the confusion matrices are to each other. ICC seems like a good tool to use here, and I could do this by treating each confusion matrix as a 16-dim vector and then feeding the 16x20 array of data into an ICC algorithm. Again, which ICC method is most appropriate? 1-k?

In my niche, I don't see people use ICC much, so I don't have any experience with the method and wanted to double check with experts to prevent myself from doing something stupid.

Here are the types in my analysis software:

'1-1': The degree of absolute agreement among measurements made on randomly selected objects. It estimates the correlation of any two measurements.

'1-k': The degree of absolute agreement of measurements that are averages of k independent measurements on randomly selected objects.

'C-1': The degree of consistency among measurements. Also known as norm-referenced reliability and as Winer's adjustment for anchor points.

'C-k': The degree of consistency for measurements that are averages of k independent measurements on randomly selected objects. Known as Cronbach's alpha in psychometrics. The degree of consistency for averages of k independent measures made under the fixed levels of column factor.

'A-1': The degree of absolute agreement among measurements. Also known as criterion-referenced reliability. The absolute agreement of measurements made under the fixed levels of the column factor.

'A-k': The degree of absolute agreement for measurements that are averages of k independent measurements on randomly selected objects. The degree of absolute agreement for measurements that are based on k independent measurements made under the fixed levels of the column factor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.