Assessing predictive accuracy on longitudinal data

Question

Suppose you have a longitudinal dataset, in which several subjects, sampled independently of each other, were measured at one or more timepoints each. The timepoints need not have equal intervals between them, and different subjects may have been assessed at different timepoints. In general, there are lots of subjects and only a few measurements per subject. You'd like to validate the predictive accuracy of a statistical model on all observations. What's a good way to do this?

To me, it seems that when the model is making a prediction for subject $s$ at time $t$, it should be allowed to use as training data the observations from all subjects other than $s$ and from all of $s$'s observations that occurred earlier than $t$. This leads to a procedure similar to leave-one-out cross-validation except that future timepoints for the same subject are also excluded. (Edit: This seems to be the procedure adopted by Rao, 1987.) It's not obvious how to adapt this idea to 5- or 10-fold cross-validation, although I'd prefer that because it's much faster.

Here's the research project I'm working on now that's making me think about this. I'm looking at how drug use can predict HIV-related behavior among homosexually active men in the mSTUDY. Considering, in particular, a dichotomous measure of risky sexual behavior, which I'm considering only in HIV-negative subjects, the breakdown on the dependent variable is 106 risky versus 260 safe. Here's the number of subjects with each number of observations available:

 Observations   Number of subjects
 1                51
 2                52
 3                45
 4                19

Besides the subject identifier and timepoint, I have as the predictor variables 13 different numeric drug-use variables. I'll probably want to add demographic variables at some point.

Rao, C. R. (1987). Prediction of future observations in growth curve models. Statistical Science, 2, 434–447. doi:10.1214/ss/1177013119

Answer 1

To me, it seems that when the model is making a prediction for subject $s$ at time $t$, it should be allowed to use as training data the observations from all subjects other than $s$ and from all of $s$'s observations that occurred earlier than $t$.

This approach may (may!) make you overestimate your accuracy, depending on your actual use case. And that in two related ways. Why?

1. For the first reason, consider the following situation. Assume you have multiple weather stations and different weather parameters $W_{s,t}$ measured at each station $s$ and time $t$, like temperature, cloud cover, wind speed & direction. The goal is to predict the weather at all stations.

   When predicting the weather at station $s$ for time $t$, would you consider it valid to use all weather data from other stations than $s$ and the weather data up to time $t$ for station $s$?

   I hope not. Because the weather $W_{s',t}$ at time $t$ (and later) at stations $s'\neq s$ may be highly correlated with the weather $W_{s,t}$ at station $s$ and time $t$ you want to predict. So using $(W_{s',t})_{s'\neq s}$ may lead to data leakage. After all, you will likely also want to predict $W_{s',t}$ for some $s'\neq s$ using your model - would it be licit to use $W_{s,t}$ here?

   What you should be using is the weather data up to but not including $t$ for all stations, i.e., $(W_{s',t'})_{s';t'<t}$.

   Of course, the same holds in your application. The behavior for subject $s$ at time $t$ may be highly correlated with the behavior of other subjects $s'\neq s$ at the same time $t$, for all kinds of social reasons.

2. At some time $t'$, you are predicting for subject $s$ at time $t$. If $t>t'$, then you don't know the information for the other subjects at time $t$ yet! (In the example above, we would be forecasting tomorrow's weather at New York City - but then we wouldn't know tomorrow's weather in Boston, either!) Thus, it doesn't make sense to use the information from other subjects at a time $t$ (or even later!) that we have not observed yet.

Actually, both points may be irrelevant if we are interpolating in the time domain. For instance, we may be sitting in Boston and wonder what the weather right now is in New York City. Then it of course makes sense to look outside and see what the weather is right now outside the window, together with what we know of past weather in New York City. In this situation, you would be explicitly using the fact of a high correlation between the weather at these two cities, and the fact that you know the weather at Boston at the exact time when you want to "predict" it in NYC.

Thus, the takeaway is that you should tailor your model to the question you are actually trying to answer, and to the information set you have at the time you run your predictions in a "production" environment (assuming this notion makes sense).

This is a common setup in time series forecasting, and there called "time series cross validation". (I personally find this term a bit unfortunate and prefer "holdout set" and similar.) In my own applications, for instance, we often want to include the impact of weather on retail sales, and here the second point above comes in: if we want to predict tomorrow's sales, then we don't know tomorrow's weather yet and need to predict this in turn, which adds noise.