2
$\begingroup$

We have a dataset where multiple measurements of the same variable are taken from individuals and we want to measure the intraobserver reliability. However, the individuals each come from different regions, and our question is about whether the different regions affect this variable. Can we use a regular ICC to assess the intraobserver reliability here, or should we be using a special version that ignores the between-region variance, or some-other-way takes into account that the data has a hierarchical structure? I'm not sure how best to think about this.

I've tried Googling, but ICCs are so naturally intertwined with hierarchical models anyway, that none of what I've found so far seems relevant.

Thanks a lot!

Improve this question
$\endgroup$
3
  • $\begingroup$ Why not remove Region effect first and before the estimation of ICC? I.e. compute ICC on the residuals left after predicting all the pool of responces by Region. $\endgroup$
    – ttnphns
    Commented Jul 24, 2018 at 10:58
  • $\begingroup$ That is, repetition reliability is likely not to depend on region, region is just a level-shifting "distractor". $\endgroup$
    – ttnphns
    Commented Jul 24, 2018 at 11:00
  • $\begingroup$ @ttnphns Thank you so much for fast response and apologies for my slow reply -- genuine reasons -- the residuals idea is brilliant, I wish I had thought of that. However, I'm still struggling with whether it is really a "distractor" or if it is not in fact the key thing we're trying to measure? If our questions is whether region affects these numbers, is that component not the core component that we are actually trying to measure? But I'm struggling with whether that logic makes sense... $\endgroup$
    – justme
    Commented Aug 1, 2018 at 10:58

1 Answer 1

Reset to default
0
$\begingroup$

I came across a similar question and found this post:

Adjusted ICC with mixed effects model in R

The rpt function in R has the following function: "A wrapper function for (adjusted) repeatability estimation from generalized linear mixed-effects models fitted by restricted maximum likelihood (REML). Calls specialised functions depending of the choice of datatype and method." With repeatability they refer to ICC in the article.

There are 2 packages, rpt and rptr, both seem convenient.

The appropriate method is explained in https://www.ncbi.nlm.nih.gov/pubmed/20569253

Hope this may help.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.