Working with the scikit-learn library for python, consider a linear regression model such as the elastic net (
Furter assume that one wishes to work with a normalised feature space whatever the reasons. Two options naturally come to mind:
ElasticNetobject with the
normalizeattribute set to true (should one simultaneously set the
fit_interceptattribute, he/she should make sure it is not set to false in which case the
normalizeargument would be ignored, see relevant docstring)
Pipelineconsisting of a
Normalizer(pre-processing method) and an
normalizeattribute set to false.
These approaches are similar. However, it seems like the user community tends to prefer the second option.
This is because when cross-validation is applied to a pipeline object rather than a model object, for instance through
cross_val_score(pipe, X, y), the feature space preprocessing is part of the full learning process (i.e. is applied appropriately for each CV fold).
Now, suppose that instead of working with the 'naive' elastic net, one were to work with an elastic net whose hyper parameters are determined by cross-validation (
ElasticNetCV class for instance).
In that case, option 2 above does not seem to be the right way to go. More specifically, since the normaliser is fitted on the training set, when we'll work through the internal cross-validation (hyper-parameter determination), we will work with folds that have been normalised using data that is not part of the fold, which is typically data snooping.
In otherwords, the pipeline way of doing things seems fine for simple cross-validation but could be dangerous for nested cross-validation since it could produce optimistically biased cross-validation scores.
Can someone confirm this or am I missing something?