# Classifying handwritten digits using PCA

Classify handwritten digits using PCA. Use 200 digits for the train phase and 20 for the test.

I have no idea how PCA works as a classification method. I've learned to use it as a dimension reduction method where we subtract the original data from its mean, then we calculate the covariance matrix, eigenvalues and eigenvectors. From there, we're able to choose principal components and ignore the rest. How should I classify a bunch of handwritten digits? How to distinguish data from different classes? Or does it mean something totally different, that I must use PCA for feature extraction purposes and use a classification method afterwards?

• I crossposted the question here to receive hints about its matlab code. Apart from that, I need ideas as to what should I do about the problem in general. – Gigili Jan 30 '13 at 19:28
• Why do you have to use PCA? Is this a homework problem? Please specify. Otherwise, why do you have to restrict yourself to using PCA? – user765195 Jan 31 '13 at 3:27
• The answer in the linked thread on StackOverflow is better than the ones here. – amoeba says Reinstate Monica Jan 28 '15 at 23:40

PCA gives you the directions in feature space along which the variance of the data is maximal, i.e. containing most information about your data (if you assume it to be Gaussian distributed).

Another way to look at it, and the one that best applies here is to see them as the vectors which give you the best reconstruction of your date in terms of the Euclidean distance. There are a number of ways to derive it. There are many books and tutorials where you can find those derivations. I would recommend you reading this one.

What you would do is to calculate the first $k$ components for each of the character classes. WHen presented a new sample, you calculate the projection of that sample onto each of the principal components (PC) of each class, and take as resulting class the one giving the highest projection, in other words, the one that the sample is closest to.

Last but not least, PCA is rather employed as a dimensionality reduction procedure than as a classifier. For that there are much better approaches. Mixture of probabilistic PCAs, SVMs, and lot more.

You should use PCA for feature selection before applying a classifier. This usage of PCA in classification has grown common since a 1991 paper on classifying faces using "eigenfaces". The procedure is basically:

1. Select k orthogonal dimensions of maximum variation (given by the eigenvectors of the covariance matrix) for all the data together (after normalization).
2. Project all data points (training and test) onto these k dimensions to get k-dimensional feature vectors.
3. Learn and classify the k-dimensional feature vectors using your favorite classification method.

The role of PCA in this scheme is to reduce the complexity of the data handled by the classification stage, and thus perhaps afford stronger classifiers. One failure case for this scheme is when there is large intra-class variation in orthogonal directions to the inter-class variation, and thus the inter-class directions are lost. In practice though, the scheme is very commonly used and found to be successful for a variety of problems.

I know of one way, there might be others.

The probabilistic formulation of PCA, pPCA, is actually a density model-it allows you to estimate $p(x)$. Thus, in a generative approach you can get a predictive distribution via Bayes rule: $p(c|x) \propto p(x|c) p(c)$. Thus, you can do one PCA for each handwritten digit. For a new digit, pick the class for which the reconstruction error of the corresponding PCA (which is proportional to the likelihood) is lowest.