The Intra-class Correlation Coefficient measures the degree to which a categorical variable accounts for variability in a continuous variable. It is often used to index the degree of clustering in a dataset.

learn more… | top users | synonyms (1)

0
votes
0answers
18 views

Correct approach for computing interrater reliability

everybody. Can't find a strong answer to this "special case" elsewhere on the interwebs, so I'm hoping you can help. I want to computer interrater agreement (actually, in this case, inter-method ...
2
votes
1answer
70 views

Notation of Variance of Residuals in Multilevel Modeling

I am having some trouble to understand the notation of variance of residuals in multilevel modeling . In this paper "Sufficient Sample Sizes for Multilevel Modeling" , in p.87 below equation (3) , ...
1
vote
1answer
21 views

acceptable ICC for 6-point Likert scale

I have pairs of pictures of participants making facial expressions (240 pairs per participant, 11 participants). Two raters agreed on a criterion and rated each one of the pairs on a scale from 1 to ...
1
vote
1answer
26 views

Calculating ICC using random intercept and residual variances

I have seen many posts refer to the calculation of the ICC by utilizing the variance of the random intercept and the residual variance from a multilevel model. Does anyone have this formula handy?
0
votes
0answers
23 views

Interpretation of random-effects GLS regression (multilevel model, 2 levels, strictly hierarchical)

There are several values in this analysis that I wonder how to interpret, and whether they can be used to calculate some sort of effect size. Contextual information: level 1 trials in a test - level ...
0
votes
0answers
11 views

How do you show adequate inter-rater agreement?

I have data from 230 individuals from different companies, who rated their perceptions of their individual companies' organizational climates (four different scales ranging from 7 questions to 4 ...
1
vote
0answers
34 views

Intra-class correlation and ANOVA for multi-class classification?

I have trained and evaluated with leave-one-out cross-validation classifiers (SVM, logistic regression, deep belief network, gaussian naive bayes, k-nearest neighbour) on my dataset. My dataset ...
0
votes
0answers
6 views

ICC using one-way random model

I need to compute the ICC value on my data set. I have a set of videos rated by different raters; each rater rated 4 videos randomly chosen. So, I would like compute ICC by using one-way random model ...
3
votes
1answer
28 views

The valid variance components to include in a generalizability theory model (anova) involving nesting? Are these correct interactions?

My question is about generalizability theory. A study by Bergeron, Floyd, McCormack, and Farmer (2008) is only adding to this confusion. Basically, in G-theory you want the first model you run to ...
0
votes
0answers
40 views

Intraclass Correlations ICC1, ICC2, ICC3?

I am doing some research for my teacher training. I am looking at the agreement between student-assessment and teacher-assessment. Essentially there are 4 tasks. Students scored the task and I ...
0
votes
0answers
48 views

Interpreting summary of ICC results in R package “psych”

I have a dataframe "all.st.icc", where 27 subjects have measures of social behavior for each of 8 different years. First four rows of data are as follows: ...
0
votes
0answers
92 views

ICC interpretation: low/moderate/high

Correlation coefficients and effect sizes often have 'rules of thumb' on how to interpret their values. These guidelines (see for example http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/effectSize) ...
0
votes
0answers
18 views

Help: testing repeatability within a group, and conducting a PCA for observations within those perhaps-repeatable variables?

I'm trying to analyze behavioral observations of nesting birds within and across various field sites. I know that certain behaviors, such as latency to flight, amount of vocalizations, and defensive ...
1
vote
0answers
26 views

Two separate or a single mixed-effects model?

Lets say my data is exam results of students nested in schools and there are two kinds of schools: private and public ones. Now, what I am interested is the variance between private and public ...
1
vote
0answers
112 views

Intraclass Correlation Coefficient value range vs strength of association

I read on this site that ICC is a good measure of association(effect size) between nominal and interval data..and the significance test suggested was anova...My question is, given the association is ...
0
votes
0answers
87 views

within group agreement, intra-class correlation, one way ANOVA to justify data aggregation

I am working on my research involving aggregating individual data into group data. Predictor: Task conflict (3-item scale) Process conflict (3-item scale) ...
0
votes
0answers
77 views

Which Intraclass Correlation Coefficient for Repeated Measures by a Single Rater

I have a set of measurements (up to 4 per patient, with some missing values) made on 500 patients made by the same physician (a simple example is given below). ...
0
votes
0answers
222 views

sample size for test-retest reliability

I am interested in estimating the following: test-retest reliability for ordinal items; test-retest reliability for interval-level scale scores consisting of the items; and the optimal sample size ...
1
vote
1answer
16 views

interrater reliability for proportional (not count) data

My experiment has the following form: a rater assesses a visual image and records a single value, p, which is his estimate of the proportion of that image composed of a single color. Thus, the data ...
0
votes
0answers
28 views

ICC for growth model

I have searched quite a bit for this answer. Is there an ICC for unconditional growth models. Not the unconditional means model, but a growth model. I suppose I mean to ask is there a way to describe ...
0
votes
0answers
21 views

Observer agreement in multilevel binary data

I have a set of participants indicating with drawn circles where they experience pain on a manikin of the human body. These drawings were then independently rated by two human raters: each of 29 body ...
0
votes
1answer
43 views

Panel data: Compare two different assessment methods

Just wondering if anyone has any inventive (but relatively simple) ideas about how to approach comparing panel data from two different assessment methods that collect the same variables. Basically, I ...
1
vote
0answers
54 views

Intraclass correlation and simulating data

I will apologize in advance for this somewhat messy problem. I am trying to take an ICC reported from a previously published manuscript and design a general simulation program. Let's say the ...
2
votes
1answer
464 views

Interpretation of variance in multilevel logistic regression

Please help me to interpret the findings of my model. The specifications of the model are: Dependent variable: treatment (1) or no-treatment (0). Independent variables: age, number of drugs used, ...
1
vote
1answer
70 views

$R^2$ for mixed models = ICC?

I will be referring here to Nakagawa and Schielzeth (2013). As those authors state, $R^2$ for OLS regression could be defined as follows: $$R^2 = \frac{\sum^n_{i=1}(\bar{y} - ...
4
votes
1answer
490 views

Acceptable values for the intraclass correlation coefficient (empty model)

I'm using xtmixed in Stata to test a Hierarchical Linear Model. My problem is that variance at level 2 is about 4% of the total variance. So most of the variance is ...
0
votes
0answers
15 views

Intraclass correlation with a sparse rater-subject matrix

I'm working on a problem pertaining to the reliability of faculty evaluations of students. My dataset contains ratings of ~160 students given by ~50 faculty members. Most students are rated by 2 ...
0
votes
0answers
90 views

Adding in L2 predictor in GLiMM fails even though ICC is high

I'm using SPSS GLiMM to run a repeated measures logistic regression model. Research Qn I'm interested to know if a MP is more likely to make a particular response (e.g., apology, challenge, denial) ...
0
votes
0answers
72 views

How do you aggregate individual scores to a team level using ICCs?

I'm pretty sure that at this point I have read every article and forum post that has to do with aggregating scores to a team level. I've learned how to do the some of the analyses, like R(wg) and ...
0
votes
0answers
48 views

type of ICC coefficient to use in my study for inter und intrarater-reliability

I am new to the subject and not sure, which ICC I have to use in order to analyse inter/intrarater-reliability of my measurements. A physician examined 649 healthy individuals with an ...
0
votes
0answers
7 views

Comparing TrueScore with the System Score

I have collected three human raters scores for an essay question(N=54). I am building Automated Scoring Algorithem. So I need to compare my machine score with the Human rater score. All are using 0-10 ...
3
votes
1answer
299 views

Multilevel logistic regression with a random slope(s)

I would like to specify a two-level logistic regression model with random intercept and random slope. Dependent variable: hospitalization (1) or no-hospitalization (0). Independent variables: age, ...
1
vote
0answers
282 views

Intraclass Correlation Coefficient in mixed model with random slopes

I have the following model m_plot fitted with lme4::lmer with crossed random effects for participants (...
3
votes
1answer
467 views

How to interpret glmer results (variance, correlation and ICC)

I'm a beginner in statistics and I have to run multilevel logistic regressions. I am confused with the results as they differ from logistic regression with just one level. I don't know how to ...
0
votes
0answers
59 views

ICC to assess independence of observations in a cluster randomized trial

How can I compute an ICC to assess independence of observation assumption using SPSS for ANOVA. The data on two DVs were collected from participants one time over 28 weeks in 28 clusters (so this is ...
1
vote
0answers
67 views

What are the tests to measure agreement between two methods of measurement when the unit of measurement is different between the two methods?

Bland-Altman plot measure the agreement between two different methods which measures the same variable. As far as I understand Bland-Altman can measure only if the two methods have the same unit of ...
1
vote
0answers
94 views

Intraclass Correlation Coefficient Question

I have two datasets in which I probably need to use the intraclass correlation coefficient, and I want to make sure I choose the right type. Dataset 1: I had 20 randomly selected human subjects rate ...
0
votes
1answer
62 views

Classification using correlation

Given two correlation matrix (each $p \times p$), where each belongs to a different group, is it possible to classify a new sample into one of the group (based on the correlation matrix only)? What ...
2
votes
1answer
127 views

How to test whether ICCs from different groups are significantly different?

I need to compare two generalizability (G) coefficients for data that are from two separate populations. G coefficients are a type of intraclass correlation coefficient (ICC). The literature on ...
0
votes
0answers
216 views

Should I use both paired t-test and intraclass correlation coefficient to examine the interrater reliability of an instrument?

As title, I am assessing the interrater reliability of a questionnaire (actually a Barthel Index (BI)) performed by four assessors (A,B,C,D) on about 25 test subjects. I plan to examine the IRR from ...
0
votes
0answers
25 views

Interpreting the relInterIntra {irr} results in R

I am using this for a reliability analysis test for quantitative values between two raters. I got my results using the "relInterIntra {irr}" package in R, but the reported values gave me several ...
0
votes
0answers
76 views

Non-normal data (ceiling effect) and intraclass correlation

I am wondering if there is any way of doing an intraclass correlation, two-way, mixed, with non-normal data (in this case a ceiling effect)? Alternatively, what the interpretation (if any) would be? ...
0
votes
0answers
31 views

How to analyse individual consistency from three non-normally distributed variables?

I would like to know if individuals were consistent across three different days. I measured the latency of individuals for a maximum of ten minutes. Many individuals had a latency of 0 and some ...
1
vote
1answer
1k views

Can ICC values be negative?

I am calculating the ICC as per the method of Shrout-Fleiss (1979), using the (3, 1) model in which the judges are fixed as are the targets. Some of my ICC values are negative. Is this to be ...
4
votes
1answer
204 views

Likert scale inter rater agreement

What is the best inter rater agreement test for Likert scale type questions. As far I saw that cronbach alpha is for internal consistency, or it shows how good items are related to describe the main ...
3
votes
2answers
436 views

Multilevel models including random slopes: how to calculate variance

In a linear mixed model, you take the covariance between data into account by adding a random intercept per cluster. For example, you measure the effect of a drug campaign over time on students, and ...
1
vote
0answers
100 views

Correlated predictors in linear model: where does shared variance go?

I analyze a dataset consisting of responses to two conditions with a multiple regression analysis. I dummy code the two conditions to retrieve the contribution of each of them to the data. Up to here, ...
0
votes
0answers
140 views

Measure of accuracy or measure of agreement for continuous data with true values

I have two continuous variables. Variable $X$ represents the true values (basically it is atmospheric pressure measured via a barometer). Variable $Y$ represents a way to approximate the atmospheric ...
2
votes
1answer
35 views

Statistic to find out association between ordinal data

I have a 4x3 contigency table with ordinal data for a research finding in tuberculosis, I analysed cytology patterns and afb patterns; cytological pattern $A,B,C,D$ (independent variable) and afb ...
0
votes
0answers
59 views

What is a “subject” in ICC?

I have been working in R and have finally got some code working regarding intraclass correlation coefficients. In the output I get something called "subjects". I thought it was to do with the number ...