Usually while working on data preparation I split the data into train and test sets, fit PCA, clustering etc on the train set and apply the transformations on the test set. So the test set doesn't get involved in PCA, clustering or any other process and hence prevents leakage.
While working with cross-validation, how should I prepare my data? For each fold that I hold out, should I fit PCA, etc on the remaining folds and then apply transforms to the holdout? Or, should I pre-process the data in one go?
If I fit PCA, etc for every fold, that would mean I am running the data preparation pipeline several times and that becomes a huge bottleneck for larger datasets. If I process it one go, I am introducing leakage and might get optimistic results.
What should one do in such cases?