I am fairly new to the machine learning, and I have been going over all the great posts about cross-validation today and I have a question regarding PCA and cross-validation, I don't have enough points to comment on the PCA and train/test split so I thought maybe I should post a new question.
My understanding is the most correct procedure is the following which I saw in an earlier post:
for each fold:
split data
conduct PCA on the 90% used for training
pick the number of components
fit linear regression
predict the 10% held out
end
My main question is, if I want to do eigenfaces with PCA and SVM I would split up my set of images into my training and validation sets, and then apply the PCA to each new split in my cross-validation and optimization? My confusion comes because I was following an example on Scikit-learn where they divide the data, and then proceed to caluclate the PCA for the split data. Next, they run GridSearchCV which I understand is doing a cross-validation to tune the parameters. Does this tuning introduce a bias because it preprocessed the data or is it okay for some reason? I have attached the relevant sections of the example below.
###############################################################################
# Split into a training set and a test set using a stratified k fold
# split into a training and testing set
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25)
###############################################################################
# Compute a PCA (eigenfaces) on the face dataset (treated as unlabeled
# dataset): unsupervised feature extraction / dimensionality reduction
n_components = 150
print("Extracting the top %d eigenfaces from %d faces"
% (n_components, X_train.shape[0]))
t0 = time()
pca = RandomizedPCA(n_components=n_components, whiten=True).fit(X_train)
print("done in %0.3fs" % (time() - t0))
eigenfaces = pca.components_.reshape((n_components, h, w))
print("Projecting the input data on the eigenfaces orthonormal basis")
t0 = time()
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)
print("done in %0.3fs" % (time() - t0))
###############################################################################
# Train a SVM classification model
print("Fitting the classifier to the training set")
t0 = time()
param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5],
'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1], }
clf = GridSearchCV(SVC(kernel='rbf', class_weight='auto'), param_grid)
clf = clf.fit(X_train_pca, y_train)
print("done in %0.3fs" % (time() - t0))
print("Best estimator found by grid search:")
print(clf.best_estimator_)
X_test_pca
. Personally, I think it is a bit unfortunate that the code is not completely separated into training code and testing code (i.e. I'd move applying the PCA to the test data to the very bottom of the code and then go on with testing the SVM). $\endgroup$