# Questions tagged [intraclass-correlation]

Intraclass correlation coefficient (ICC) measures how much quantitative values are similar or associated within groups. There exist several versions and alternative formulations of ICC. The correlation is used to assess a dataset clusteredness, agreement among raters and in other settings.

257 questions
Filter by
Sorted by
Tagged with
23 views

### Using and Reporting ICC for Aggregated Data About Groups of People

I've designed a survey that consists of multiple questionnaires with Likert-scale questions, each of which targets a specific variable in my research model. The participants of this survey are ...
16 views

### How to decompose variance into intra and inter group variance for random effect

I have difficulty understanding something that may be something simple. It concerns the decomposition of variance between inter and intragroup variance. Here is how I understand it so far: If I have a ...
• 211
15 views

### In a 1-way ANOVA, is the square-root of the coefficient of determination the intra-class correlation?

The actual question I mean to ask is if the square-root of the effect size is the intra-class correlation (ICC). When the 1-way ANOVA test is viewed as a bivariate analysis between a categorical ...
• 3,863
23 views

### appropriate ICC

My apologies this is a rookie question, I am aware of what ICC(Intraclass correlation) is, and that it measures how strongly subjects in the same group resemble each other. However I am not sure how ...
12 views

### The meaning of "ICC corrected by age and comorbidity is best described in detail"

I received feedback on a manuscript comparing the mortality of two populations. Both crude and adjusted mortality rates are compared. Adjusting was done for age and comorbidity score and it is clearly ...
• 1,533
74 views

### Computing ICCs in Matlab, to assess rater consistency (inter-rater agreement)

In an experiment, participants rated each of several musical stimuli on 5 aspects (ratings A to E), across ~3,000 trials. Stimuli were performed (by a previous set of subjects) either solo or duo (...
• 1,548
1 vote
15 views

### Which intraclass correlation coefficient (ICC) to use to verify the test-retest reliability of a questionnaire?

I want to check the test-retest reliability of a questionnaire. -A total of 300 adolescents from the same school (10 classes) participated in the study. -The instrument was applied in two moments with ...
• 61
1 vote
12 views

### ICC in Random intercept model with a predictor variable

Are we able/ or is it correct to calculate ICC in random intercept model which has a preditor variable within? Does this value give us the variation of the outcome which is purely related to subject ...
• 11
28 views

### Calculate intra-class correlation (or some other measure of reliability) with variable number of raters

I have some stimuli rated by users. I want to estimate how much of the variance in ratings is due to the stimuli, so that I can compare between various types of ratings which are the most reliable. ...
76 views

### Is this documentation for the R 'psych' package ICC() function incorrect?

I am not a statistician, but from my reading about various types of intraclass correlation measures, ICC(2,1) (Shrout and Fleiss nomenclature) is a measure of absolute agreement sensitive to additive ...
• 145
17 views

### Is there a formula expressing the relationship between standard deviation, standard deviation of cluster effect and standard deviation within cluster?

Suppose I have data on child test scores of children from different schools. Children are the unit of observation and the schools are clusters. The standard deviation of the test score variable is 1.5....
• 13
1 vote
40 views

### ICC vs Repeated Measures ANOVA to Justify Multilevel Model

I have heard of using ICCs to determine if there is variability to justify using a multilevel (i.e., mixed effects, hierarchical) model. However, I've recently seen studies examine whether there is a ...
• 21
20 views

### Intraclass correlation coefficiant for underdispersed count data

I would like to calculate the intraclass correlation coefficiant (ICC). My data are counts, approximately normal distributed and the variance is smaller than the mean. If I am not mistaken, this ...
• 11
12 views

### Test-retest reliability with varying number of measurements

I developed a new method to measure a certain quantity on a radiological image of a patient. I have available varying numbers of studies per patient. I applied the method to all images available for ...
1 vote
23 views

### What is the best way to compare fixed and random effects of a GLMM?

I am doing a research in which i am trying to measure the importance of the doctor that is in charge of a patient in a medical decision. For that (and others reasons) i have used a GLMM using lmer4 ...
14 views

### Estimating confidence intervals for ICCs in a multilevel Poisson model

Can anyone help me with this in R? I need the conf. intervals for the ICC in a glmmTMB model. I know I should either go for bootstrapping or something Raykov 2015 ...
1 vote
19 views

### How to conduct a meta-analysis on ICCs (i.e., test-retest reliability)?

I have calculated two-way single consistency ICCs in R using the irr package. There are multiple ICCs representing test-retest reliability across these time points. Is there a way to compute a meta-...
• 309
1 vote
53 views

### What is the correct nesting structure for calculating the Inter-rater reliability or intra class correlation (ICC) with multiple raters?

I need to figure out which inter-rater reliability for multiple raters I should calculate. I'm trying to figure out how I should nest or structure certain observations within my dataset in order to ...
98 views

### Alternative to Intra-class correlation coefficient (ICC) for continuous data

I am working on a research project that involves a reliability study. In this case, 3 raters have assigned two sets of scores to 50 items: an ordinal score on a scale from 1 - 3, and a continuous ...
97 views

### Difference between Multilevel I^2 and ICC?

I have been using the excellent metafor package for a work-related project and for some reason there's a concept that I can't wrap my head around. In his guide (...
1 vote
43 views

### Weighted Kappa with rating variables that have different set of categories

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT Rater1 rated the the ...
• 457
22 views

### Intraclass correlation with skewed dataset

There is data on 20 people taking the same test 5 times each to look at the repeatability of the test. I understand that intraclass correlation is usually the best test to use to look at a question ...
14 views

### crude and adjusted Intra Class Correlation (ICC)

I have a data set with four different hormones which were assessed at two time points on 40 healthy participants. For these hormones I want to analyse the intra-individual stability of plasma ...
12 views

### Assessing Consistency and Agreement of Two Groups of Raters in an ordinal dataset

I'm using a real dataset of convenience to look at one group of subjects who are being assessed for "frailty". The goal is to see whether assessments by non-experts are comparable to experts....
• 1
1 vote
157 views

### Disagreement between ICC and paired t-test

I am trying to determine whether the values obtained by an algorithm calculation are systematically different from the the criterion values. The criterion values are obtained via manual observation ...
• 43
411 views

### How to estimate the variance of correlated observations?

Assume we have n observations $x_i$ (i from 1 to n), each from the a normal distribution with mean 0 and some variance component: $X_i \sim N(0, \sigma^2)$. The random variables $X_i$s have some (let'...
• 20k
46 views

### How to compare the test-retest reliability of two groups?

I have one group (NBD, N=23) and another group (PWA, N=22) who each have data at Test (time point 1) and retest (time point 2). I've calculated test-retest reliability using a oneway, consistency, ...
51 views

### Inter and intra rater reliability (intra class correlation) from linear mixed model

This is a follow-up question to this one. My dataset consists of: one dependent variable (value) measured 3 times (observation) ...
• 105
325 views

### How to calculate two-way mixed ICC?

I want to calculate ICC for two-way mixed model, single rater, and either for consistency and absolute agreement. I found two packages for calculating ICC (intra-class correlation coefficient), that ...
• 39
1 vote
34 views

### Compare within- and between-individuals variances with lme4?

How can I compare the variances of two random effects variables? I have an experiment with several proteins measured in individuals (ind = 4 levels) before and after a treatment (treatment = 2 levels ...
• 115
81 views

### Calculating ICC for a beta-binomial GLMM

I understand that ICC in binomial GLMMs with a logit link can be calculated via R, where the residual deviance is (pi ^ 2) / 3. However, this is assuming that the ...
• 41
1 vote
13 views

### Testing L2 effects without any L2 variance?

I'm testing a hypothesized effect of an L2 variable on L1 outcomes (students nested in classrooms), but a few of my outcome variables have no L2 (classroom-level) variance (i.e. the within-classroom, ...
• 31
56 views

### On the distribution of sample intraclass correlation for normal data

Consider i.i.d random vectors $\boldsymbol X_1,\boldsymbol X_2,\ldots,\boldsymbol X_n$ having a $p$-variate normal distribution $N_p(\boldsymbol \mu,\Sigma)$ where $\Sigma$ has compound symmetry of ...
• 9,120
43 views

### Should dimensions with low reliability be omitted from EFA?

A set of items were rated on 25 dimensions by 85 participants. I've computed the ICCs on each dimension and they are all good (> .75) except one dimension which is moderate (.69) and one that is ...
• 1,641
67 views

### Within-cluster correlation in lognormal shared frailty models

According to Peter Austin (ref. below), in a gamma shared frailty model (i.e. a Cox regression model with cluster specific random effects which are iid logarithms of gamma distributions), the within-...
• 391
79 views

### How can I use the ICC as a measure for inter-rater agreement on a set of (more or less) correlated items?

I have a question about inter-rater agreement using the ICC. I'm trying to test, whether my study participants (the raters) rate the personality of a fictional character similarly - think of a ...
65 views

### Is it correct to compare ICC/VPC values between linear and logit models?

I am currently working on a research project involving the decomposition of between-group error using the intra-class correlation coefficient (ICC) which you may understand more intuitively as the ...
19 views

### intracluster correlation coefficient: are standard errors valid?

I am working with panel data and I want to estimate the fixed-effects model. I suspect that SE might be misleading due to (possibly) the necessity of clustering. Would the intracluster correlation ...
• 27
37 views

### What's a good way to calculate agreement between examiners when the measure is binary?

I want to assess ability to identify tumor types in humans vs. a computers. There are 7 tumor types. I am interested in calculating agreement between the human examiners but I am not sure the best way ...
• 63
56 views

### ICC calculation for examiner and timepoint

How do I calculate an ICC for a dataset in which there are k examiners, and each examiner takes n measurements? I know that I can use the psych package in R to calculate the ICC across the k*n ...
• 63
104 views

### When are circular statistics needed (reference)?

I have a set of data of measurements for the lateral angle of the internal acoustic meatus (it's an angle between the canal of our inner ear and another bone structure in our skull). The measurements ...
• 105
1 vote
47 views

### ICC for cluster within cluster (survey package R)

So I'm not quite sure about the terminology but what ever it's called the actual scenario is straight-forward. I'm dealing with a two-stage cluster sample. It's also stratified but let's set that to ...
• 934
365 views

### How to calculate "by hand" between group variance and the ICC for a three-level hierarchical model?

Introduction: This question is related to this and this questions that were answered before, but I would like a more detailed answer on how between-groups variance are calculated in a (null) three-...
• 121
245 views

### Can pearson's correlation coefficient be converted to Cohen's kappa?

I'm interested in meta-analysing the agreement between two measures. Depending on whether the variables are continuous/categorical/binary, different studies present agreement as either: A Pearson's ...
• 135
437 views

### How to interpret near-zero ICC in mixed models?

I am running an analysis of data with multiple observations per person. I assumed that therefore a mixed model analysis would be appropriate. Back then in my statistics class, we were told that ...
• 93
162 views

### Interpretation of Intraclass correlation coefficient (ICC) as a reliability estimate for hierarchical designs - lme4

I am working on the following problem and would really appreciate your help: The data set at hand contains medical ratings (scale 1-5, assumed interval) by 5 independent raters, on 7 occasions, for 60 ...
1 vote
63 views

### ICC from random effects model VS ANOVA

ICC is talked about in the context of ANOVA as well as random effects model. In case of the ANOVA approach, the calculation is defined as: ...
• 123
1 vote
192 views

### Why does the intraclass correlation coefficient (ICC) increase as covariates are added to a hierarchical logistic regression model?

I fit three Bayesian binary multilevel logistic regression models. Schematically they look like this: ...
• 389
44 views

### Crossed or random effects? (lme4)

I am examining the effects of the type of context change (change in personnel, change in setting) on response rate of a targeted behavior (e.g., aggression) among patients in a treatment center. There ...
• 51