I am a medical doctor and definitely not an expert in statistics, although I think I do okay and I am familiar with R.

I have discussed the following issue with a statistician but I am still not sure whether we came up with the right solution. I hope one of you can help.

I want to estimate the precision (repeatability) of a continuous variable measured on x-rays (PT) and evaluate the influence of different raters by specifying the variance between and within raters (inter- and intra-rater variance).

I have had 4 raters measure the continuous variable on 67 x-rays twice, thus each x-ray has been measured 8 times in total.


'data.frame':   536 obs. of  4 variables:
 ID    : Factor w/ 67 levels 
 Rater : Factor w/ 4 levels 
 Time  : Factor w/ 2 levels 
 PT    : num  40.4 29.3 36 58.8 40.5 ...

THE MODEL we came up with was a linear mixed effect fit:

a <- lmer(PT~ (1|ID) + (1|Rater:ID), data=Pre) 

From exploring the web I found that ":" is rarely used and that the model is equivalent to:

b <- lmer(PT~ (1|ID/Rater), data=Pre)

THE RESULT of either model a or b is:

Linear mixed model fit by REML ['lmerMod']
Formula: PT ~ (1 | ID/Rater)
   Data: Pre

REML criterion at convergence: 2729.2

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-5.7958 -0.2266 -0.0165  0.2214  4.8076 

Random effects:
 Groups   Name        Variance Std.Dev.
 Rater:ID (Intercept)  4.630   2.152   
 ID       (Intercept) 97.178   9.858   
 Residual              2.695   1.642   
Number of obs: 536, groups:  Rater:ID, 268; ID, 67

Fixed effects:
            Estimate Std. Error t value
(Intercept)   18.611      1.214   15.34

THE INTERPRETATION according to the statistician was that:
The residual variance represents the intra-rater variance and the inter-rater variance is the sum of Rater:ID and residual variance.

Are these interpretations valid?
My concern is especially if the residual variance is truly an expression of the intra-rater variance or if I should somehow include the "Time" variable in the model to specify the variance between the first and second measurements within raters within subjects.


First of all, the interpretation is probably correct. However, looking at other research in radiology of continuous measures, especially the paper by Gstottner (1), why don't you consider using the psych package in R(2) and the interclass correlation coefficient? Given that the ICC is used on other publications, editors might want it so that readers can compare with similar papers

1 Gstoettner M, Sekyra K, Walochnik N, Winter P, Wachter R, Bach C. Inter- and Intraobserver Reliability Assessment of the Cobb Angle: Manual Versus Digital Measurement Tools. Eur Spine J. 2007; 16(10): 1587-92.

2 Reville, W. (2013) psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, http://CRAN.R-project.org/

| cite | improve this answer | |
  • 1
    $\begingroup$ Thank you very much for a swift answer. Nice to know that we are not all wrong :) We also intend to report ICC as a measure of reliability, however as I understand the ICC it is an estimate of how well the continuous measure is at distinguishing between different subjects. We also want to estimate the measurement error per subject and thus estimate how much of a difference between two measurements on the same subject –say 1 yr apart– would represent a true change. $\endgroup$ – Dennis Jul 15 '15 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.