I've recently learned about the wonderful PCA and I've done the example outlined in scikit-learn documentation.
I am interested to know how I can apply PCA to new data points for classification purposes.
After visualizing PCA in a 2 dimensional plane (x,y axis), I see that I can probably draw a line to separate the data points so that one side would be of one classification and the other of another classification. How do I draw this "boundary" and apply this to the new data points?
PCA is a dimension reduction tool, not a classifier. In Scikit-Learn, all classifiers and estimators have a predict method which PCA does not. You need to fit a classifier on the PCA-transformed data. Scikit-Learn has many classifiers. Here is an example of using a decision tree on PCA-transformed data. I chose the decision tree classifier as it works well for data with more than two classes which is the case with the iris dataset.
from sklearn.decomposition import PCA
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
# load data
iris = load_iris()
# initiate PCA and classifier
pca = PCA()
classifier = DecisionTreeClassifier()
# transform / fit
X_transformed = pca.fit_transform(iris.data)
classifier.fit(X_transformed, iris.target)
# predict "new" data
# (I'm faking it here by using the original data)
newdata = iris.data
# transform new data using already fitted pca
# (don't re-fit the pca)
newdata_transformed = pca.transform(newdata)
# predict labels using the trained classifier
pred_labels = classifier.predict(newdata_transformed)
SciKit learn has a convenient tool called Pipeline which lets you chain together transformers and a final classifier:
# you can make this a lot easier using Pipeline
from sklearn.pipeline import Pipeline
# fits PCA, transforms data and fits the decision tree classifier
# on the transformed data
pipe = Pipeline([('pca', PCA()),
('tree', DecisionTreeClassifier())])
pipe.fit(iris.data, iris.target)
pipe.predict(newdata)
This is especially useful when doing cross-validation as it prevents you from accidentally re-fitting ANY step of the pipeline on your testing dataset:
from sklearn.cross_validation import cross_val_score
print cross_val_score(pipe, iris.data, iris.target)
# [ 0.96078431 0.90196078 1. ]
By the way, you may not even need to use PCA to get good classification results. The iris dataset doesn't have many dimensions and decision trees will already perform well on the untransformed data.
If you want to apply PCA to new data, you must have fit a model first on some training dataset. What is the model you will ask? This is the mean vector you subtracted from the dataset, the variances you used to "whiten" each data vector and the learned mapping matrix. So in order to map new dataset in the same space as the training data, you first subtract the mean, whiten it and map it with the mapping matrix.
