When using cross validation, shouldn't it be mandatory to pipeline the preprocessing steps? I'm currently experimenting with sklearns transformer classes in combination with pandas. During my application on a kaggle data set and being picky on not choosing information of the test data, I realized the following:
Typically, in those challenges you have a training data set and a test data set. We perform preprocessing and say we impute the missing values in the train and test data set by the mean of the training data. Then we do our usual stuff of finding a valid model.
But wait, let's forget the test data set for a moment (on kaggle you don't even have values for the response variable on the test set). We want to estimate the generalizability of our model, this can be achieved by using cross validation on the training data. But on each testing fold, we potentially imputed missing data, i.e. we have information of the corresponding training set!
For example imputing by mean will yield different values on the whole training set and each cv fold.
So, shouldn't data preprocessing steps always be included in the pipeline?
Can this be a severe problem?
 A: 
So, shouldn't data preprocessing steps always be included in the pipeline?

Ideally, it should. It's part of the model after all. A case can be made when you have lots of data, when the estimates across folds end being almost the same. But in general it's better to assess the whole model building process performance, and that includes preprocessing when it takes into account training data.

Can this be a severe problem?

With big data probably not. With small data (and include this categorical variables with a lot of levels), this can potentially add a huge bias to your predictions. It's mostly due to the fact that your estimates are of lower quality when you only use the train data, and the estimates might jump a lot across folds, degrading your model.
A: In addition to @Firebug's answer,


*

*As for imputation for the test data set: IMHO it is legitimate to use training data to do this: This would be an "imputation model" trained on training data and applied to test data, and in principle not different from deriving, say, the mean from training data and then applying this e.g. for centering to the test data as well.
However, you need to make sure no test data leaks into the training set: the test data needs to stay unknown.

*From my application-centered point of view, pre-processing is part of the model building and needs to be included in the validation (just like hyperparameter tuning is), i.e. also needs to be re-calculated inside the cross validation loop.  

*Some preprocessing steps can be pulled out of the validation loop without risking a data leak:   


*

*the step in question calculates strictly within each case (i.e. does not consider any other cases but the one that is processed at the moment), and

*there was no preprocessing step that considered more than one case up to this point. 



In other words, the cross validation loop needs to start before the first (pre)processing step that uses more than one case.
If you need a citation for this, we wrote it e.g. in
Dochow, S.; Beleites, C.; Henkel, T.; Mayer, G.; Albert, J.; Clement, J.; Krafft, C. & Popp, J.: Quartz microfluidic chip for tumour cell identification by Raman spectroscopy in combination with optical traps. Anal Bioanal Chem, 2013, 405, 2743-2746.
DOI: 10.1007/s00216-013-6726-3 
