Combining PCA, feature scaling, and cross-validation without training-test data leakage

Question

The sci-kit learn documentation for cross-validation says the following about using feature-scaling and cross-validation:

Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction

I understand the reason behind this is to prevent information leakage between the training & test sets during cross-validation, which could result in an optimistic estimate of model performance.

I am wondering then, if I wish to use Principal Component Analysis to reduce the size of a feature set before training say a regression model, and PCA requires feature-scaling to be effective, how do I chain feature-scaling to PCA to cross-validated regression, without introducing data leakage between the train-test splits in the cross-validation?

Answer 1

You need to think feature scaling, then pca, then your regression model as an unbreakable chain of operations (as if it is a single model), in which the cross validation is applied upon. This is quite tricky to code it yourself but considerably easy in sklearn via Pipelines. A pipeline object is a cascade of operators on the data that is regarded (and acts) as a seemingly single model confirming to fit and predict paradigm in the library.

Answer 2

For the benefit of possible readers who don't use the scikit pipeline:

• Centering and scaling the training subset does not only result in the centered and scaled training data but also in vectors describing the offset and scaling factor. When predicting new cases, this offset and scale is applied to the new case, and the resulting centered and scaled data then passed to the principal component prediction
• which in turn applies the rotation determined from fitting the training data.
• and so on, until the final prediction is reached.

Answer 3

For anyone who might stumble upon this question, I have a solution using scikit-learn's Pipeline, as recommended in the accepted answer. Below is the code I used to get this to work for my problem, chaining together StandardScaler, PCA and Ridge regression into a cross-validated grid-search:

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([("scale", StandardScaler()),
                 ("reduce_dims", PCA()),
                 ("ridge", Ridge())
                ])


param_grid = dict(reduce_dims__n_components = [0.5, 0.75, 0.95],
                  ridge__alpha = np.logspace(-5, 5, 10),
                  ridge__fit_intercept = [True, False],
                 )

grid = GridSearchCV(pipe, param_grid=param_grid, cv=10)
grid.fit(X, y)

Answer 4

I've encounter some problems with pipelines (for example, if I want to apply my own custom function, it is a real hazard) so here is what I use instead:

X_train, X_test, y_train, y_test = train_test_split(X, Y, stratify=Y, random_state=seed, test_size=0.2)
sc = StandardScaler().fit(X_train)
X_train = sc.transform(X_train)
X_test = sc.transform(X_test)
pca = PCA().fit(X_train)
X_train = pca.transform(X_train)
X_test = pca.transform(X_test)
eclf = SVC()
parameters_grid = {
                'C': (0.1, 1, 10)
                   }
grid_search = GridSearchCV(eclf, parameters_grid, cv=cv, refit='auc', return_train_score=True)
grid_search.fit(X_train, y_train)
best_model = eclf.set_params(**grid_search.best_params_).fit(X_train, y_train)
test_auc_score = roc_auc_score(y_test, best_model.predict(X_test))

I realize it is a bit long but it is clear on what you are doing.