# Applying PCA to test data for classification purposes

I've recently learned about the wonderful PCA and I've done the example outlined in scikit-learn documentation.

I am interested to know how I can apply PCA to new data points for classification purposes.

After visualizing PCA in a 2 dimensional plane (x,y axis), I see that I can probably draw a line to separate the data points so that one side would be of one classification and the other of another classification. How do I draw this "boundary" and apply this to the new data points?

• PCA isn't a classifier, but it is possible to place new observations into the PCA assuming the same variables used to "fit" the PCA are measured on the new points. Then you just place the new points at the weighted sum of the variable scores (loadings), weights given by the data. That said, arbitrarily drawing a line through your PCA doesn't sound like a good choice of classifier to me... – Reinstate Monica - G. Simpson Apr 1 '15 at 20:28

PCA is a dimension reduction tool, not a classifier. In Scikit-Learn, all classifiers and estimators have a predict method which PCA does not. You need to fit a classifier on the PCA-transformed data. Scikit-Learn has many classifiers. Here is an example of using a decision tree on PCA-transformed data. I chose the decision tree classifier as it works well for data with more than two classes which is the case with the iris dataset.

from sklearn.decomposition import PCA
from sklearn.tree import DecisionTreeClassifier

# initiate PCA and classifier
pca = PCA()
classifier = DecisionTreeClassifier()

# transform / fit

X_transformed = pca.fit_transform(iris.data)
classifier.fit(X_transformed, iris.target)

# predict "new" data
# (I'm faking it here by using the original data)

newdata = iris.data

# transform new data using already fitted pca
# (don't re-fit the pca)
newdata_transformed = pca.transform(newdata)

# predict labels using the trained classifier

pred_labels = classifier.predict(newdata_transformed)


SciKit learn has a convenient tool called Pipeline which lets you chain together transformers and a final classifier:

# you can make this a lot easier using Pipeline

from sklearn.pipeline import Pipeline

# fits PCA, transforms data and fits the decision tree classifier
# on the transformed data
pipe = Pipeline([('pca', PCA()),
('tree', DecisionTreeClassifier())])

pipe.fit(iris.data, iris.target)

pipe.predict(newdata)


This is especially useful when doing cross-validation as it prevents you from accidentally re-fitting ANY step of the pipeline on your testing dataset:

from sklearn.cross_validation import cross_val_score
print cross_val_score(pipe, iris.data, iris.target)
# [ 0.96078431  0.90196078  1.        ]


By the way, you may not even need to use PCA to get good classification results. The iris dataset doesn't have many dimensions and decision trees will already perform well on the untransformed data.

• Perhaps it is important to note that PCA can be useful even if the dimensionality is not reduced. You can have a dataset of dimensionality $d$ and the discriminative directions are the ones which correspond to maximum variance. You keep the $d$ dimensions but the basis are different. Mapping your data on those new basis will help to discriminate classes better than in original basis. – Vladislavs Dovgalecs Apr 1 '15 at 20:25
• @xeon I didn't know that. – Austin Richardson Apr 1 '15 at 20:35
• Perhaps it is better to see if you imagine PCA as a rotation. If it happens that your dataset has this property such that the classes can be discriminated by variance, then this rotation will be all you need. – Vladislavs Dovgalecs Apr 1 '15 at 20:37
• @xeon: When all dimensions are kept, the only thing that one achieves by PCA is decorrelating the dataset. It can indeed be beneficial for some classifiers, but the vast majority does not care. – amoeba says Reinstate Monica Apr 1 '15 at 20:42
• @amoeba I completely agree, this is just a small detail. I had to deal with such particular dataset and always remember that lesson. – Vladislavs Dovgalecs Apr 1 '15 at 20:43

If you want to apply PCA to new data, you must have fit a model first on some training dataset. What is the model you will ask? This is the mean vector you subtracted from the dataset, the variances you used to "whiten" each data vector and the learned mapping matrix. So in order to map new dataset in the same space as the training data, you first subtract the mean, whiten it and map it with the mapping matrix.