How to choose train, test and validation data for two-staged experiments?

Question

I have a question concerning the possibility of the train-test-validation split in a staged experiment setup.

The data I used is split up into 3 parts: train, test and validation data.

Then I try to run a two-staged experiment setup: In the first stage I examine several data preparation steps. In the second stage I use the best data preparation method from the first stage to continue with different model-setups (like a partial model, etc.). In both stages I do a k-fold cross-validation on the training dataset and evaluate the results with the test dataset. Only in the second stage I evaluate the final results using the validation dataset. The procedure is shown below.

I am now asking myself whether this is a possible approach or am I getting too much data leakage from the first stage to the second, as I am performing the cross-validation on the same training dataset in both cases?

Answer 1

There are two distinct, but related reasons why you should not do this:

• From the testing perspective, many pre-processing techniques and inparticular any data-driven tuning in the first stage will become a leak for subsequent splitting, yes. (Or, require all data that was in that stage to be training data later on; you can do that and acquire separate test data subsequently)

• From the model building perspective, data pretreatment/preprocessing and modeling typically interact: the best model (choice of algorithm and hyperparameters) can depend on the pretreatment.
  E.g., data may become linear due to preprocessing, or it may become non-linear. I've seen data where more complex pre-tretreatment leads to less complex modeling later on (in a way that one may say that the overall complexity is the same)

  You thus in general cannot optimize pretreatment independently of later modeling steps.

  There are exceptions to this in that sometimes it is possible to treat some decisions separately from other parts of the modeling process. However, this is something that requires detail knowledge on application question, data, pretreatment process and the "actual" modeling. It should be a deliberate decision, and I'd recommend to spell out the reasoning why it is appropriate.

  E.g., I work with data where the physics behind the data acquisition process means that particular and well-known artifacts occur. These artifacts are completely independent of the measured samples. In that case, I can justify to set up a data quality filter to detect and either remove or correct those artifacts. This can be done in a way that does not influence other properties of the data. The validation of the final model can then include how well those decisions work.