I have a measurement protocol that seeks to characterise the visual sensitivity of a subject. I would like to assess how reliable the measurements from this procedure are.
I have measured a small number of subjects, with two repeated measurements per day for a number of days (not all subjects did the same number of days). The data are similar in form to this (code for R):
# generate some example data: dat <- expand.grid(subject=1:4,rep=c(1,2),day=seq(0,28,by=4)) # extra subject with more days: d <- expand.grid(subject=5,rep=c(1,2),day=seq(0,36,by=4)) dat <- rbind(dat,d) dat$subject <- factor(dat$subject) dat$y <- rnorm(nrow(dat),mean=3,sd=0.25)
which produces some data whose summary looks like:
subject rep day y 1:16 Min. :1.0 Min. : 0.00 Min. :2.455 2:16 1st Qu.:1.0 1st Qu.: 8.00 1st Qu.:2.817 3:16 Median :1.5 Median :16.00 Median :2.970 4:16 Mean :1.5 Mean :14.95 Mean :2.978 5:20 3rd Qu.:2.0 3rd Qu.:24.00 3rd Qu.:3.141 Max. :2.0 Max. :36.00 Max. :3.617
The measurement value is ratio-scaled. Day is relative to the other days tested within a subject (i.e. not all subjects' "day N" corresponds to the same actual day).
One simple measure of consistency is to correlate the scores within a day - a simple test-retest measurement. However, this doesn't usefully use the information that the same subject repeated the measurement across days. Another measure that I considered was an autocorrelation of the measurements as a function of time, but this is extremely noisy because not all subjects did the same number of days, so it must be done for each subject.
I also modelled the data using a linear model with "days" and "rep" as continuous predictors, in a Bayesian hierarchical framework. This shows that the slope for day and the slope for repetition within a day are not credibly different from zero. This indicates that the test scores are fairly stable across days and repetitions.
However, one thing I would like to be able to do is to directly compare the variance between subjects with the variance within a day or across days within subjects. That is, to test whether the variability of measurement error + within subject fluctuations is smaller than the variability in normal visual sensitivity across individuals.
Does anyone have any suggestions for how to model this?