How to split training, testing and validation data for human subject experiment I have experimental data of 25 human subjects. What is the efficient way to split it into train, test, and validation? It is given that,the number of data is different for different subjects. This means subject n1 might have a large volume of data where n2 might have few samples of data.
I am using four-fold validation with 5 subject data kept as testing and the rest of 20 as training and validation. The problem is splitting validation data from these 20 subjects. I can assume randomly take two or three participant data as validation. However, some participant has a large number of the data point. If anyway those participant is selected as validation then validation data will have a chance to greater or equal to the training set, which does not make sense.
Is there any better approach? or should I fix the validation subject for each fold?
 A: Firstly, what is the aim? If you are trying to predict something for a new (no previous data on them) subject, then splitting by subject makes sense. If you are trying to predict what happens with a subject later based on early data from a subject, then splitting within subject by time makes sense.
Secondly, stratified cross-validation splits can be a useful idea to keep something (e.g. here amount of data - for stratification perhaps split into 2 - 4 categories or so - alternatively looping over all or randomly trying some possible CV splits and picking one(s) with approx. balanced data amounts across folds in terms of, say, the mean amount of data or some slightly robust-to-outliers type of mean) approximately balanced across CV folds. It's also worth asking ourselves, why we want this balance. I assume the concern is that, if for some folds the validation set is really small, then the validation score for that fold might be rather random, which may be inefficient? One could of course reduce that variability also by simply doing repeated-K-fold CV (depending on how expensive the model fitting is).
Thirdly, this is a pretty low number of subjects, so bootstrapping could also be an attractive option (depending on how expensive the model fitting is). See e.g. this discussion of the topic.
