I have coded a PCA by hand and am trying to make sense of the transformation. As a simple example, lets suppose my dataset has 2 instances with three features each:

>Instance #1: 2 3 4
>Instance #2: 2 4 1

These three features are of course with respect to the traditional X, Y, Z axes.

Suppose during PCA I find two significantly large eigenvalues corresponding to two vectors ( 1 1 2) and (1 2 3). By multiplying these vectors with each of my instances, I would get the following new features in a smaller space;

>Instance #1: 13 20
>Instance #2: 8  13

For example, 13 is computed from (2*1) + (3*1) + (4*2).

Now, here is my problem. When feeding these new features into a classifier, they are just numbers -- the classifier is not aware of them being along some direction (i.e., the classifier does not see the eigenvectors, it only sees the new features, which are the projections of the data on these eigenvectors).

Typically when demonstrating the benefits of PCA, one is shown a plot in which each number is a magnitude along a given direction. In those plots, one is easily able to see the benefits of PCA as the direction information helps to show how the numbers are located in the space. But when inputting the data to a classifier, the classifier is not aware of these directions, we only feed in the array of principal components (not eigenvectors), so I get confused on how the classifier exploits the direction information that it never sees...


You can understand PCA's main purpose with a simple example. Consider that your two variables $X$ and $Y$ are perfectly correlated i.e. all data points lie on a line passing through the origin. By performing PCA, you rotate the data to the direction of maximum variance. This will result in a data that has one constant dimension: the new rotated data (in the coordinates of principal components) will be a horizontal straight line located at $x = 0$.

As you understand, this transformation will make classification an easier task since linear data correlations have been taken out of the data. After PCA, you can train your classifier on only one dimension instead of two (as you would in the original dataset) without any loss of information.

  • $\begingroup$ +1 The classifier never knows what directions mean, we (often) do. To the classifier, they're just numbers, a vector of these numbers (say N numbers) that correspond to a sample define a point in an N-dimensional space. Note that the numbers you feed to a classifier don't have units, so how could the classifier know anything about what directions mean? The downside is if you as a human want to interpret coefficients, etc, from your classifier. At that point you would care about units (dollars, inches, age, age-squared, etc) but you don't have any from PCA. $\endgroup$
    – Wayne
    Oct 22 '15 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.