0
$\begingroup$

I am fairly new to the machine learning, and I have been going over all the great posts about cross-validation today and I have a question regarding PCA and cross-validation, I don't have enough points to comment on the PCA and train/test split so I thought maybe I should post a new question.

My understanding is the most correct procedure is the following which I saw in an earlier post:

for each fold:
    split data
    conduct PCA on the 90% used for training
    pick the number of components
    fit linear regression
    predict the 10% held out
end

My main question is, if I want to do eigenfaces with PCA and SVM I would split up my set of images into my training and validation sets, and then apply the PCA to each new split in my cross-validation and optimization? My confusion comes because I was following an example on Scikit-learn where they divide the data, and then proceed to caluclate the PCA for the split data. Next, they run GridSearchCV which I understand is doing a cross-validation to tune the parameters. Does this tuning introduce a bias because it preprocessed the data or is it okay for some reason? I have attached the relevant sections of the example below.

###############################################################################
# Split into a training set and a test set using a stratified k fold

# split into a training and testing set
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25)


###############################################################################
# Compute a PCA (eigenfaces) on the face dataset (treated as unlabeled
# dataset): unsupervised feature extraction / dimensionality reduction
n_components = 150

print("Extracting the top %d eigenfaces from %d faces"
      % (n_components, X_train.shape[0]))
t0 = time()
pca = RandomizedPCA(n_components=n_components, whiten=True).fit(X_train)
print("done in %0.3fs" % (time() - t0))

eigenfaces = pca.components_.reshape((n_components, h, w))

print("Projecting the input data on the eigenfaces orthonormal basis")
t0 = time()
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)
print("done in %0.3fs" % (time() - t0))


###############################################################################
# Train a SVM classification model

print("Fitting the classifier to the training set")
t0 = time()
param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5],
              'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1], }
clf = GridSearchCV(SVC(kernel='rbf', class_weight='auto'), param_grid)
clf = clf.fit(X_train_pca, y_train)
print("done in %0.3fs" % (time() - t0))
print("Best estimator found by grid search:")
print(clf.best_estimator_)
$\endgroup$
4
  • 1
    $\begingroup$ I don't understand why this is not a duplicate of "PCA and train/test split". Can you please clarify your confusion? Your conceptual understanding is correct: PCA is done on the training data and then the same transformation is applied to the trainining and testing data. This seems to be exactly what is done in the scikit code your pasted here. So what's the problem? $\endgroup$
    – amoeba
    Commented Jan 16, 2015 at 13:34
  • 1
    $\begingroup$ I agree with @amoeba. The example code is missing the validation of the optimized SVM, which however can be done with X_test_pca. Personally, I think it is a bit unfortunate that the code is not completely separated into training code and testing code (i.e. I'd move applying the PCA to the test data to the very bottom of the code and then go on with testing the SVM). $\endgroup$
    – cbeleites
    Commented Jan 16, 2015 at 14:55
  • 1
    $\begingroup$ @amoeba, I was concerned that because gridsearchCV is performing a 3-fold cross-validation that there was a problem that with performing the PCA outside of gridsearchCV. However after thinking about it some more, I think you are correct in that there is no problem because the code is not attempting to cross-validation to estimate the performance of the model. Instead, we are just using our test data to measure the performance at the end. cbeleites, you also have a good point about the way the code is structured. I can add the SVM validation to the example, or should I delete the post? Thanks! $\endgroup$
    – drj
    Commented Jan 16, 2015 at 19:15
  • $\begingroup$ @drj: No, don't delete your post! I would suggest that you update it with a short update that answers your own confusion (and/or perhaps extends the code -- as you wish). And we will mark it as a duplicate of "PCA and train/test split". Your question might help somebody in the future. $\endgroup$
    – amoeba
    Commented Jan 16, 2015 at 21:02

0

Browse other questions tagged or ask your own question.