Suppose you have a longitudinal dataset, in which several subjects, sampled independently of each other, were measured at one or more timepoints each. The timepoints need not have equal intervals between them, and different subjects may have been assessed at different timepoints. In general, there are lots of subjects and only a few measurements per subject. You'd like to validate the predictive accuracy of a statistical model on all observations. What's a good way to do this?

To me, it seems that when the model is making a prediction for subject $s$ at time $t$, it should be allowed to use as training data the observations from all subjects other than $s$ and from all of $s$'s observations that occurred earlier than $t$. This leads to a procedure similar to leave-one-out cross-validation except that future timepoints for the same subject are also excluded. (Edit: This seems to be the procedure adopted by Rao, 1987.) It's not obvious how to adapt this idea to 5- or 10-fold cross-validation, although I'd prefer that because it's much faster.

Here's the research project I'm working on now that's making me think about this. I'm looking at how drug use can predict HIV-related behavior among homosexually active men in the mSTUDY. Considering, in particular, a dichotomous measure of risky sexual behavior, which I'm considering only in HIV-negative subjects, the breakdown on the dependent variable is 106 risky versus 260 safe. Here's the number of subjects with each number of observations available:

 Observations   Number of subjects
 1                51
 2                52
 3                45
 4                19

Besides the subject identifier and timepoint, I have as the predictor variables 13 different numeric drug-use variables. I'll probably want to add demographic variables at some point.

Rao, C. R. (1987). Prediction of future observations in growth curve models. Statistical Science, 2, 434–447. doi:10.1214/ss/1177013119

  • $\begingroup$ Just to be sure I understood, you want to build a classifier (response is binary) where the predictors are the 13 drug-use values, the subject id and time? However, if time is a predictor, thus would mean that the response depends on it, i.e., that a subject who is safe at time $t$ may become risky later or vice versa. Is this true? $\endgroup$ – DeltaIV Oct 18 '16 at 6:26
  • 1
    $\begingroup$ @DeltaIV Yes, that's all correct. $\endgroup$ – Kodiologist Oct 18 '16 at 6:46
  • $\begingroup$ Survival Analysis might be useful here. $\endgroup$ – Firebug Oct 24 '16 at 1:53
  • 1
    $\begingroup$ @Firebug I'm not familiar with survival methods, but I think they wouldn't apply here because neither state is permanent (i.e., a risky subject can become safe and vice versa). $\endgroup$ – Kodiologist Oct 24 '16 at 18:23
  • $\begingroup$ I was wondering what you decided to use for this? Trying to solve a similar problem. $\endgroup$ – user0 Jan 7 '17 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.