I will create an example to explain my question. Say I have two independent data clusters in a 3D space (x,y,z). The two data clusters, Cluster A and B, are very distant to each other, such that running PCA will results in the first component explaining this distance. These two clusters also have different magnitude of "clusterness".
I ran PCA to find whether Cluster A or B is more clustered, and plotted PC1 vs. PC2. As stated before, PC1 explained the physical distance between Cluster A and B. Along PC2, I found that Cluster A data exhibited a linear variation, while Cluster B data are clustered together.
By using this fact alone, can I conclude that since the data of Cluster A are differentiated in PC2, then Cluster B is relatively more tightly clustered than Cluster A?
I understand that PC1, PC2 and PC3 are orthogonal. However, try to look at a 3D problem. If two different clusters have the same length in PC1, it means PC1 is not useful to distinguish which of the two is more clustered. But in PC2, the less clustered group will always be explained by PC2 (because by definition PC2 explains more variance than PC3). Hence, in this case PC3 is not needed because PC1 and PC2 are sufficient to infer this observation.
What if we extend it beyond 3D? If I have hundreds of variables, will PC1 vs. PC2 still be able to distinguish the magnitude of "clusterness" of two indepedent data sets?
For example, this is a PC1 vs. PC2 plot of data with hundreds and thousands of variables. The eigenvalue for PC2 is 3.5, while for PC3 is 2.2.
Just from the above picture, am I allowed to conclude that the right group is more clustered than the left group?