Suppose you have a longitudinal dataset, in which several subjects, sampled independently of each other, were measured at one or more timepoints each. The timepoints need not have equal intervals between them, and different subjects may have been assessed at different timepoints. In general, there are lots of subjects and only a few measurements per subject. You'd like to validate the predictive accuracy of a statistical model on all observations. What's a good way to do this?
To me, it seems that when the model is making a prediction for subject $s$ at time $t$, it should be allowed to use as training data the observations from all subjects other than $s$ and from all of $s$'s observations that occurred earlier than $t$. This leads to a procedure similar to leave-one-out cross-validation except that future timepoints for the same subject are also excluded. (Edit: This seems to be the procedure adopted by Rao, 1987.) It's not obvious how to adapt this idea to 5- or 10-fold cross-validation, although I'd prefer that because it's much faster.
Here's the research project I'm working on now that's making me think about this. I'm looking at how drug use can predict HIV-related behavior among homosexually active men in the mSTUDY. Considering, in particular, a dichotomous measure of risky sexual behavior, which I'm considering only in HIV-negative subjects, the breakdown on the dependent variable is 106 risky versus 260 safe. Here's the number of subjects with each number of observations available:
Observations Number of subjects 1 51 2 52 3 45 4 19
Besides the subject identifier and timepoint, I have as the predictor variables 13 different numeric drug-use variables. I'll probably want to add demographic variables at some point.
Rao, C. R. (1987). Prediction of future observations in growth curve models. Statistical Science, 2, 434–447. doi:10.1214/ss/1177013119