I see that in PCA the first principal component maximizes the variances amongst all the points within the data set. What exactly does this mean, what does it show and what does every other principal component thereafter tell me?
I read these really nice simple explanation on PCA that helped me understand PCA as a whole:
To decide how many eigenvalues/eigenvectors to keep, you should consider your reason for doing PCA in the first place. Are you doing it for reducing storage requirements, to reduce dimensionality for a classification algorithm, or for some other reason? If you don't have any strict constraints, I recommend plotting the cumulative sum of eigenvalues (assuming they are in descending order). If you divide each value by the total sum of eigenvalues prior to plotting, then your plot will show the fraction of total variance retained vs. number of eigenvalues. The plot will then provide a good indication of when you hit the point of diminishing returns (i.e., little variance is gained by retaining additional eigenvalues).
However, it's still unclear to me as to really what each PC means and what PC2 now represents and PC3, etc.