I have coded a PCA by hand and am trying to make sense of the transformation. As a simple example, lets suppose my dataset has 2 instances with three features each:
>Instance #1: 2 3 4 >Instance #2: 2 4 1
These three features are of course with respect to the traditional X, Y, Z axes.
Suppose during PCA I find two significantly large eigenvalues corresponding to two vectors ( 1 1 2) and (1 2 3). By multiplying these vectors with each of my instances, I would get the following new features in a smaller space;
>Instance #1: 13 20 >Instance #2: 8 13
For example, 13 is computed from (2*1) + (3*1) + (4*2).
Now, here is my problem. When feeding these new features into a classifier, they are just numbers -- the classifier is not aware of them being along some direction (i.e., the classifier does not see the eigenvectors, it only sees the new features, which are the projections of the data on these eigenvectors).
Typically when demonstrating the benefits of PCA, one is shown a plot in which each number is a magnitude along a given direction. In those plots, one is easily able to see the benefits of PCA as the direction information helps to show how the numbers are located in the space. But when inputting the data to a classifier, the classifier is not aware of these directions, we only feed in the array of principal components (not eigenvectors), so I get confused on how the classifier exploits the direction information that it never sees...