Modeling measurement error (reliability?) using repeated measurements I have a measurement protocol that seeks to characterise the visual sensitivity of a subject. I would like to assess how reliable the measurements from this procedure are.
I have measured a small number of subjects, with two repeated measurements per day for a number of days (not all subjects did the same number of days). The data are similar in form to this (code for R):
# generate some example data:
dat <- expand.grid(subject=1:4,rep=c(1,2),day=seq(0,28,by=4))

# extra subject with more days:
d <- expand.grid(subject=5,rep=c(1,2),day=seq(0,36,by=4))
dat <- rbind(dat,d)

dat$subject <- factor(dat$subject)

dat$y <- rnorm(nrow(dat),mean=3,sd=0.25)

which produces some data whose summary looks like:
 subject      rep           day              y        
 1:16    Min.   :1.0   Min.   : 0.00   Min.   :2.455  
 2:16    1st Qu.:1.0   1st Qu.: 8.00   1st Qu.:2.817  
 3:16    Median :1.5   Median :16.00   Median :2.970  
 4:16    Mean   :1.5   Mean   :14.95   Mean   :2.978  
 5:20    3rd Qu.:2.0   3rd Qu.:24.00   3rd Qu.:3.141  
         Max.   :2.0   Max.   :36.00   Max.   :3.617 

The measurement value is ratio-scaled. Day is relative to the other days tested within a subject (i.e. not all subjects' "day N" corresponds to the same actual day).
One simple measure of consistency is to correlate the scores within a day - a simple test-retest measurement. However, this doesn't usefully use the information that the same subject repeated the measurement across days. Another measure that I considered was an autocorrelation of the measurements as a function of time, but this is extremely noisy because not all subjects did the same number of days, so it must be done for each subject.
I also modelled the data using a linear model with "days" and "rep" as continuous predictors, in a Bayesian hierarchical framework. This shows that the slope for day and the slope for repetition within a day are not credibly different from zero. This indicates that the test scores are fairly stable across days and repetitions. 
However, one thing I would like to be able to do is to directly compare the variance between subjects with the variance within a day or across days within subjects. That is, to test whether the variability of measurement error + within subject fluctuations is smaller than the variability in normal visual sensitivity across individuals.
Does anyone have any suggestions for how to model this?
 A: You should take a look at latent-state-trait theory (Steyer et al., 1999). LST is an advancement of classical test theory. The basic idea is that with psychological measurements you are always measuring a person in a specific situation, and that this situation does in any case have an influence on the person. Therefore, neglecting the situation would distort the results of the measurement. This seems appropriate for your problem, since you have two measurements at different points in time. 
Modelling a measurement in this way will give you a decomposition of the raw scores into an estimate for the underlying factor, visual sensitivity (the latent trait), an estimate for situation specific visual sensitivity (the latent state variable), an estimation for the error variances of the state variable (state residual) and the usual measurement error/variance term. 
If $Y$ is the raw score, $\xi$ is the latent trait variable, $\tau$ is the latent state variable and $\zeta$ is the latent state residual, then in addition to the reliability $$Rel(Y_{it}) = \frac{Var(\tau_{it})}{Var(Y_{it})},$$ you can get an estimate for the consistency of the latent trait: $$Con(\xi_{it} = \frac{Var(\xi_{it})}{Var(Y_{it})},$$ as well as an estimate for the occasion specifity of a measurement: $$Spe(Y_{it}) = \frac{Var(\zeta_{it})}{Var(Y_{it})}$$.
This kind of analysis is dome using confirmatory factor analysis and also allows for testing different nested models against each other, which seems to be of interest to you.
Edit:
I just noticed that I overlooked the fact that you don't have the same number of days for each subject. Confirmatory factor analysis using LST might still work, as there are methods that can deal with missing values. I am specifically thinking of Full Information Maximum Likelihood. However, I am not sure, since I never tried this.
References
Steyer, R., Schmitt, M., & Eid, M. (1999). Latent state-trait theory and research in personality and individual differences. European Journal of Personality, 13(5), 389–408.
