I see that in PCA the first principal component maximizes the variances amongst all the points within the data set. What exactly does this mean, what does it show and what does every other principal component thereafter tell me?

I read these really nice simple explanation on PCA that helped me understand PCA as a whole:

To decide how many eigenvalues/eigenvectors to keep, you should consider your reason for doing PCA in the first place. Are you doing it for reducing storage requirements, to reduce dimensionality for a classification algorithm, or for some other reason? If you don't have any strict constraints, I recommend plotting the cumulative sum of eigenvalues (assuming they are in descending order). If you divide each value by the total sum of eigenvalues prior to plotting, then your plot will show the fraction of total variance retained vs. number of eigenvalues. The plot will then provide a good indication of when you hit the point of diminishing returns (i.e., little variance is gained by retaining additional eigenvalues).

However, it's still unclear to me as to really what each PC means and what PC2 now represents and PC3, etc.


There are already many answers for this, but here is a unique anecdotal example that might help. More explanations the merrier.

Imagine a 3D ellipsoid, like an American football, the question is to find the longest cross section, or the largest ellipse possible. Next question is, what is largest cross section possible, that is perpendicular to the first cross-section.

enter image description here

| cite | improve this answer | |
  • 1
    $\begingroup$ You might want to point out what the 3d elipsoid you describe represents in the PCA... connect the dots for the OP. $\endgroup$ – Alexis Nov 12 '16 at 7:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.