Split data into test, training and validation when some patients have multiple observations

Question

I have a dataset that has multiple observations for some patients. Patients are labelled by patient IDs and I would like to split the data into testing, training and validating groups as I am trying to create a predictive model for the dependent variable in each observation. I was using 'sample' but realised this doesn't stop one patient from appearing in more than one set. Fairly new to using R and I've looked around but can't find, or think, of a way to do this. Sample size is almost 2000, but only 1600 unique patients and I can't create weighted means of the variables in the observations reduce the size to 1600 for the split. Is anyone able to help our or suggest anything?

Answer 1

I would partition the ~400 patients with duplicates into their own analysis, since the remaining data set will have ~1200 subjects that don't have duplicates. Nearly three times as many subjects don't have repeated outcomes, and therefore the bulk of your analysis doesn't involve longitudinal outcomes. If your focus is solely on the ~400 with duplicates, you'd obviously need to find a method that will simultaneously accommodate both cross-validation and longitudinal regression with repeated measurements.

What comes to mind is a reviewer asking you why you attempted to perform a repeated measures analysis with CV when only 1/4 of ~1600 subjects had repeated measurements.

Answer 2

Grouped splitting is essential especially if you are using very flexible machine learning algorithms like a random forest. How much it makes sense compared to selecting just one observation per group depends very much on the aim of your analysis.

From a technical perspective, this can e.g. be solved like this.

If your dataset df has a column ID, one option is to use my splitTools package and write something like

ids <- splitTools::partition(
  df$ID,
  p = c(train = 0.6, valid = 0.2, test = 0.2),
  type = "grouped"
)

train <- df[ids$train, ]
valid <- df[ids$valid, ]
test <- df[ids$test, ]

60% of the IDs will be sampled into train, 20% into valid and 20% into test.