Train-test split and CV with time series data

Question

Let's say I have time series data.

'person':['A','A','A','B','B','B','C,'C','C']
'weight':[120, 123, 135, 140, 150, 151, 120, 120, 121]
'height':[5, 5, 5, 6, 6, 6, 4.5, 4.5, 4.5]
'running_time':[60,61,63,34,50,55, 60, 70, 80]
'week':[1, 2, 3, 1, 2, 3, 1, 2, 3}

Let's assume the dataset is much larger than that, of course. Let's assume I want to generate a model that will use person, weight, height, and week to predict running time (this is just an example, let's forget about other better ways to do this).

For a train test split or cross-validation, I could completely randomly split the data, where some of person A's measurements will be in train, and some in test. Or I could randomly split based on people. In other words, 70% of people go into train, 30% into test.

What would be the best way to do this?

Answer 1

Depending on your desired output, this is either a question about learning in time series, or a question about multiple instance learning. This will strongly inpact your experimental design.

If your set up is "I have this set of people and their data up to week t and I want to predict their running time in week t + 1 to week t + h" then this is a time series problem. In this case you are trying to use past performance of a particular person to predict future performance. In this case you should use time series cross-validation to evaluate your model. In time series cross-validation, you pretend to go back in to the past to date t - n and only give your model the training data that would have been available at that time (data from week t - n and earlier). You then evaluate the model for week (t - n + 1) : (t - n + h). This is repeated for the desired number of folds of cross-validation.

If your set up is "I have data for one group of people and their run times in weeks 1 : t and I want to predict what the performance of a new group of people will be in weeks 1 : t" this is an example of multiple instance learning. In this case you would want to take the second approach you mentioned where you make your cross-validation folds keeping all of the data for an individual together as training or testing for each fold. In this case information about what person is being evaluated can strongly impact the model (ex. person A always runs about 60) so including other information about the same person in the training data could keep the model from generalizing to a new person.

A good rule of thumb for cross-validation (or training/testing splits) is you should try and make you CV folds look like mini versions of the experiment you want to run. Anything that will be new when the model is implemented should be new in the validation data. If your time points will be new, you need to divide the data by time. If the people will be new, you need to divide it up by people.

Answer 2

you already explained very well the difference between MIL and Timeseries. Based on your explanation here

Anything that will be new when the model is implemented should be new in the validation data. If your time points will be new, you need to divide the data by time. If the people will be new, you need to divide it up by people.

let me give you an example :

let say "I have this set of people and their data up to week t and I want to predict what the performance of a new group of people will be in week t + 1 to week t + h", so it means we have new people with new time point then what could be a better way to devide the data to not be biased by people ?