0
$\begingroup$

I'm currently calculating an intraclass correlation coefficient for my data. As far as I'm concerned, the best choice (following this approach and the approach kindly recommended on the previous post answer) is:

Model: 2-way random effects ## two teachers' assessment of the same group of students on an exam (it could be any other pair of teachers randomly selected, thus I believe it's generalizable)
Type: average ## we'll be using the two-teacher's average in further analysis
Definition: absolute agreement

  • code:
### check if variance is homogeneous:
car::leveneTest(data$RATER1 ~ data$RATER2) ##I don't know if it's necessary, but I've seen that ICC relies on the assumption that variance is homogeneous

### run it:
library(irr)
icc(data[,2:3], model = "twoway", type = "agreement", unit = "average")

That being said, I've seen elsewhere some comments on using only 20 to 25% of the sample to calculate ICC, such as in this article and in another internet post, but not nowhere else. Why is that?

Question:

  • 1 Is it ok to use the whole sample or Should I only perform it with 20 to 25% of the data? Thanks in advance.
Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In the article you cite (Tong et al. 2022), production recordings codings (i.e., the DV) were coded by either author 2 or author 3 (i.e., each production recording was initially coded by a single coder). Because the authors are interested in calculating rater agreement, such a data set is not sufficient because each production recording was only rated once by one of the two coders. Instead of simply having each coder code all production recordings (which I assume would take a considerable amount of time), the authors instead just randomly selected 25% of the production recordings to be coded by both coders.

Now, to answer your question, you do NOT need to sample 20%, 25%, or any % of your data to calculate the ICC. You are not looking to collect additional data, as (I assume) you are just looking to calculate the ICC with the data you have.

Refrences

Tong, S. X., Tsui, R. K., Law, N. S. H., Fung, L. S. C., Chiu, M. M., & Cain, K. (2022). “And this one’s juuust right!”: Prosody Predicts Reading Comprehension in Hong Kong Chinese-English Bilingual Children.

Improve this answer
$\endgroup$
3
  • $\begingroup$ ohhh, got it! crystal clear, thank you, @Preston !! Now I understood why they've made the 25% subset. Yeah, I had two teachers rate all students scores (both teachers rated all students), I don't think I'll be collecting extra data. I may open another post for this, but this is my first time calculating an ICC (the last time, i've calculated Cronbach alpha on a very similar situation, but I believe this approach is more suitable), should I always perform Levene's test on the data before calculating ICC? $\endgroup$
    – Larissa Cury
    Nov 16, 2022 at 10:43
  • 1
    $\begingroup$ I'm glad my answer helped! Personally, I do not perform Levene's test in the context of multilevel modeling (MLM), though this does not mean it is not a good idea. I just came across an article (tandfonline.com/doi/abs/10.1080/…) discussing this, and they believe Levene's test should be used more often in the context of it should be used more often. Perhaps you should ask a question about this, as CV has active users who are more knowledgeable about HLMs than I am. $\endgroup$
    – Preston Botter
    Nov 16, 2022 at 17:12
  • 1
    $\begingroup$ Also, from both the article you posted and your other CV questions, I see you are working with language assessment data (same as me!). Given this, it may make sense to look into generalizability theory (gtheory). e.g., see journals.sagepub.com/doi/full/10.1177/… $\endgroup$
    – Preston Botter
    Nov 16, 2022 at 17:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.