Data preparation for cross validation

Question

Usually while working on data preparation I split the data into train and test sets, fit PCA, clustering etc on the train set and apply the transformations on the test set. So the test set doesn't get involved in PCA, clustering or any other process and hence prevents leakage.

While working with cross-validation, how should I prepare my data? For each fold that I hold out, should I fit PCA, etc on the remaining folds and then apply transforms to the holdout? Or, should I pre-process the data in one go?

If I fit PCA, etc for every fold, that would mean I am running the data preparation pipeline several times and that becomes a huge bottleneck for larger datasets. If I process it one go, I am introducing leakage and might get optimistic results.

What should one do in such cases?

Answer 1

• Yes, you typically need to redo preprocessing such as PCA for each surrogate model.

• Preprocessing steps that do not violate independence may be pulled in out of the cross validation loop and be done only once.

  • All steps that are calculated for each case (cases being the factor for which independent splitting into folds is done) separately (= independently of all other cases) before the first step that involves multiple cases qualify.
    This are steps like row-(case-)wise normalization as opposed to normalizing the range of a variate across all cases.