What is the most appropriate way to validate prediction models with clustered data? I am attempting to develop and validate a multivariable classification model using data from 10 clinical trials. I would like guidance on the most appropriate way to validate (internally and externally) prediction models with this clustered data:
One approach in mind would be: Use 7/10 trials for model development and internal validation and evaluate the model on each of the 3 held out trials for performance estimates via external validation. - Also, can I consider the performance estimates on the 3 non-randomly held out trials as estimates based on external validation, or is this not truly external validation?. 
A second approach in mind would be to use the full data for development and for validation by doing cross-validation by leaving each study out once for validation. But in my mind, I no longer have 'external' validation (correct me if I am wrong), but have a better model using all of the data.
 A: 
What is the most appropriate way to validate prediction models with clustered data?

For internal validation, I indeed recommend cross validation splitting by clinical trial and further known influencing factors that are crossed with the trials (e.g. patients paticipating in more than one trial).  In that case, the splitting will produce 3 subsets for each "fold": CV-training and CV-test subsets are independent of each other plus there may be a do-not-use-for-this-fold set that is independent neither of the training nor of the test set.
If the other influencing factors are nested within trial, then splitting by trial will automaticall achieve independence for them as well.
Exhaustive cross validation running though all combinations of these factors may be too tedious (too many combinations), so you may consider running only a subset of the possible splits or to run a number of train/test/do-not-use splits according to those principles (set validation). 
This way, you'll keep the full information of the trials (which would be lost by pooling) and have independent splits.
Independence can anyways be ensured only in so far as you are aware of influencing factors, and this situation is in my experience fairly common.
I'd also consider the splitting strategy fairly standard in the sense that I've been using such splits at the highest level of the data structure for clustered* data for some 15 years now, i.e. since I was working with data where we had repeated measurements for each patient during my Diplom thesis.
* we've been calling this hierarchical data structure, nested would be another term that I'd prefer nowadays.

Internal vs. External Validation
In my field, analytical chemistry (and AFAIK also for clinical chemistry), the difference between internal and external validation is 
Internal validation is performed by a lab for their method. Whereas for
External validation (method proficiency), the validation is done by an organizer outside the lab in question, for example by participating in round robin tests where an organizer sends blinded samples to the lab and then compares the lab's results to the ground truth. 
So, as you have access to the full set of studies and do training and validation, you can only perform internal validation.  To have an external validation, you'd need to find someone else to provide you with blinded samples that you predict with your final method.  
However, if the different trials are implementing the same method in different labs, i.e. you are looking at an inter-laboratory study in analytical-chemical terminology or a multicentric study in medical terminology, you may argue that while you do not have the extra blinding level an external validation would provide nor the ongoing performance estimation that repeatedly participating in a round robin provides, you do estimate an generalization error for "unknown" labs.
In any case, I recommend to clearly spell out the splitting procedure rather than assuming that everyone understands the same under the heading of internal vs. external validation. The more so, as 


*

*resampling procedures (such as cross validation, set validation or hold-out validation) belong to verification and thus can provide only a part of what is needed for full [method] validation.

*internal/external validation may spell out quite differently in different fields (e.g. a psychologist discussing internal and external validity of their method may not be concerned about different labs).

A: Let's start by defining some terms that I use/assume in my response:


*

*Test set (external validation): these sets/trials are not used in any way during training and validation (ie. parameter tuning, model selection).

*Validation set (internal validation): these sets/trials are not used during training of models, but are used for comparing different models to select one or tune parameters, etc. Read about nested cross-validation.

*Training set: these are the sets/trials whose samples you use to fit/train your model.


You can perform the task in 3 different ways each with its own pros and cons, I'm assuming you are using nested cross validation and therefore have test, validation and training set:


*

*Perform split by leaving certain number of trials in each group, as you said (3 for test, 7 for training/validation).


Advantage: If there is a single true model that is consistent across trails. Then this is the best way to make sure your model is actually modelling that.
Disadvantage: If your trails are highly heterogenous or are of different types of samples, etc. then a method performing worse at prediction does not necessarily mean it's bad.


*Merge all trials together and then leave certain percentages of samples for test and the rest for training/validation (this is what @ToddD meant).


Advantage: Simplicity.
Disadvantage: Loosing structure. Model starts picking up the heterogeneity (from what I mentioned in 1) and think that a real signal but that's just confounding. 


*Don't merge the trials and from each set take say 30% of samples for test and the rest for training/validation. So Each trial has 30% of its samples for test and the rest for training/validation.


Advantage: It could, under certain scenarios, learn the true model if for example the truth is trial/cohort specific.
Disadvantage: Possibility of picking up uninteresting trial-specific noise.
Overall, none of these approaches are best on their own. I've seen in some fields people use all of these and call them say CV1, CV2, and CV3 and then conclude based on those.
I recommend doing 1 and 3, then you can compare/contrast and come up with a conclusion.
