2
$\begingroup$

I am attempting to develop and validate a multivariable classification model using data from 10 clinical trials. I would like guidance on the most appropriate way to validate (internally and externally) prediction models with this clustered data:

One approach in mind would be: Use 7/10 trials for model development and internal validation and evaluate the model on each of the 3 held out trials for performance estimates via external validation. - Also, can I consider the performance estimates on the 3 non-randomly held out trials as estimates based on external validation, or is this not truly external validation?.

A second approach in mind would be to use the full data for development and for validation by doing cross-validation by leaving each study out once for validation. But in my mind, I no longer have 'external' validation (correct me if I am wrong), but have a better model using all of the data.

$\endgroup$
1
  • $\begingroup$ Can you simply pool the data from the trials and ignore trial membership? Analysis by trial seems like to create trial specific overfitting. $\endgroup$
    – Todd D
    Nov 7, 2019 at 22:36

2 Answers 2

4
$\begingroup$

What is the most appropriate way to validate prediction models with clustered data?

For internal validation, I indeed recommend cross validation splitting by clinical trial and further known influencing factors that are crossed with the trials (e.g. patients paticipating in more than one trial). In that case, the splitting will produce 3 subsets for each "fold": CV-training and CV-test subsets are independent of each other plus there may be a do-not-use-for-this-fold set that is independent neither of the training nor of the test set.

If the other influencing factors are nested within trial, then splitting by trial will automaticall achieve independence for them as well.

Exhaustive cross validation running though all combinations of these factors may be too tedious (too many combinations), so you may consider running only a subset of the possible splits or to run a number of train/test/do-not-use splits according to those principles (set validation).

This way, you'll keep the full information of the trials (which would be lost by pooling) and have independent splits.

Independence can anyways be ensured only in so far as you are aware of influencing factors, and this situation is in my experience fairly common.
I'd also consider the splitting strategy fairly standard in the sense that I've been using such splits at the highest level of the data structure for clustered* data for some 15 years now, i.e. since I was working with data where we had repeated measurements for each patient during my Diplom thesis.

* we've been calling this hierarchical data structure, nested would be another term that I'd prefer nowadays.


Internal vs. External Validation

In my field, analytical chemistry (and AFAIK also for clinical chemistry), the difference between internal and external validation is

Internal validation is performed by a lab for their method. Whereas for
External validation (method proficiency), the validation is done by an organizer outside the lab in question, for example by participating in round robin tests where an organizer sends blinded samples to the lab and then compares the lab's results to the ground truth.

So, as you have access to the full set of studies and do training and validation, you can only perform internal validation. To have an external validation, you'd need to find someone else to provide you with blinded samples that you predict with your final method.

However, if the different trials are implementing the same method in different labs, i.e. you are looking at an inter-laboratory study in analytical-chemical terminology or a multicentric study in medical terminology, you may argue that while you do not have the extra blinding level an external validation would provide nor the ongoing performance estimation that repeatedly participating in a round robin provides, you do estimate an generalization error for "unknown" labs.

In any case, I recommend to clearly spell out the splitting procedure rather than assuming that everyone understands the same under the heading of internal vs. external validation. The more so, as

  • resampling procedures (such as cross validation, set validation or hold-out validation) belong to verification and thus can provide only a part of what is needed for full [method] validation.
  • internal/external validation may spell out quite differently in different fields (e.g. a psychologist discussing internal and external validity of their method may not be concerned about different labs).
$\endgroup$
1
  • $\begingroup$ With this approach using nested CV, I would also like to visualize relationships & evaluate some bivariate relationships prior to modeling without biasing my performance estimate. Would this work: In fold 1, hold study 1 as the 1st outer fold, then use the remaining 9 studies (1st inner fold) to generate visualizations and evaluate bivariate associations with the outcome. Repeat for each k fold, holding out a different study as the kth outer loop, and in the remaining k-1 studies used as the kth inner loop, also generate visualizations-Giving me k sets of visualizations from the k inner loops. $\endgroup$ Dec 6, 2019 at 19:37
3
$\begingroup$

Let's start by defining some terms that I use/assume in my response:

  • Test set (external validation): these sets/trials are not used in any way during training and validation (ie. parameter tuning, model selection).
  • Validation set (internal validation): these sets/trials are not used during training of models, but are used for comparing different models to select one or tune parameters, etc. Read about nested cross-validation.
  • Training set: these are the sets/trials whose samples you use to fit/train your model.

You can perform the task in 3 different ways each with its own pros and cons, I'm assuming you are using nested cross validation and therefore have test, validation and training set:

  1. Perform split by leaving certain number of trials in each group, as you said (3 for test, 7 for training/validation).

Advantage: If there is a single true model that is consistent across trails. Then this is the best way to make sure your model is actually modelling that.

Disadvantage: If your trails are highly heterogenous or are of different types of samples, etc. then a method performing worse at prediction does not necessarily mean it's bad.

  1. Merge all trials together and then leave certain percentages of samples for test and the rest for training/validation (this is what @ToddD meant).

Advantage: Simplicity.

Disadvantage: Loosing structure. Model starts picking up the heterogeneity (from what I mentioned in 1) and think that a real signal but that's just confounding.

  1. Don't merge the trials and from each set take say 30% of samples for test and the rest for training/validation. So Each trial has 30% of its samples for test and the rest for training/validation.

Advantage: It could, under certain scenarios, learn the true model if for example the truth is trial/cohort specific.

Disadvantage: Possibility of picking up uninteresting trial-specific noise.

Overall, none of these approaches are best on their own. I've seen in some fields people use all of these and call them say CV1, CV2, and CV3 and then conclude based on those.

I recommend doing 1 and 3, then you can compare/contrast and come up with a conclusion.

$\endgroup$
3
  • $\begingroup$ I'd like to add a warning: in my field, analytical chemistry (and while I'm not clinical chemist, I've also been working with medical data) the boundary between internal and external validation is more strict than in this answer: the test set in this answer would still be considered internal validation (pending additional information about OP's data that we do not yet have). I'll try to add an answer over the weekend. $\endgroup$ Nov 8, 2019 at 12:46
  • $\begingroup$ Right, that is true in my field too. To make sure we are on the same page, testing is often done once and on data from a fully independent, unseen trial/cohort. Is this what you mean? This is exactly the reason I defined those terms at the beginning. $\endgroup$
    – NULL
    Nov 8, 2019 at 13:50
  • $\begingroup$ we go in steps: First doing the "homework" lab internally (aka internal validation). Once that looks OK, the lab is ready to participate in ring trials (external validation; the first of these commonly lead to another round of homework...). To some extent, both are ongoing to make sure the performance is still good and e.g. for clinical labs participating and passing ring trials regularly is often required by law if their results are used for patient diagnosis and treatment. What is done once are big validation studies that check e.g. limitations of the method, prove linearity etc. $\endgroup$ Nov 12, 2019 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.