In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector $X$ and its standardization $Z$ cannot be related via an orthogonal transformation. Mathematically, similar matrices (i.e. related by orthogonal transformation) have the same eigenvalues, but not necessarily the same eigenvectors.

This raises some difficulties in my mind:

1. Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)?

2. When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?

It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly), and that whenever this version cannot be used, correlation based PCA should not be used either.

I know that there is this thread: PCA on correlation or covariance? -- but it seems to focus only on finding a pragmatic solution, which may or may not also be an algebraically correct one.

In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector $X$ and its standardization $Z$ cannot be related via an orthogonal transformation. Mathematically, similar matrices (i.e. related by orthogonal transformation) have the same eigenvalues, but not necessarily the same eigenvectors.

This raises some difficulties in my mind:

1. Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)?

2. When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?

It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly), and that whenever this version cannot be used, correlation based PCA should not be used either.

I know that there is this thread: PCA on correlation or covariance? -- but it seems to focus only on finding a pragmatic solution, which may or may not also be an algebraically correct one.

In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector X and its standardization Z cannot be related via an orthogonal transform. (Mathematically) similar matrices have the same eigenvalues, but not necessarily the same eigenvectors. This raises some difficulties in my mind:

(1) Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)?

(2) When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?

It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly) and that whenever this version cannot be used, correlation based PCA should not be used either.

I'm still puzzled, hence these questions, open for answers and comments.

[I know there is thread on a similar topic, but it seems to focus only on finding a pragmatic solution, which may or may not also be a algebraically correct one.]

In principal component analysis (PCA), one can choose either the covariance matrix or the correlation matrix to find the components (from their respective eigenvectors). These give different results (PC loadings and scores), because the eigenvectors between both matrices are not equal. My understanding is that this is caused by the fact that a raw data vector $X$ and its standardization $Z$ cannot be related via an orthogonal transformation. Mathematically, similar matrices (i.e. related by orthogonal transformation) have the same eigenvalues, but not necessarily the same eigenvectors. 

This raises some difficulties in my mind:

1. Does PCA actually make sense, if you can get two different answers for the same starting data set, both trying to achieve the same thing (=finding directions of maximum variance)?

2. When using the correlation matrix approach, each variable is being standardized (scaled) by its own individual standard deviation, before calculating the PCs. How, then, does it still make sense to find the directions of maximum variance if the data have already been scaled/compressed differently beforehand? I know that that correlation based PCA is very convenient (standardized variables are dimensionless, so their linear combinations can be added; other advantages are also based on pragmatism), but is it correct?

It seems to me that covariance based PCA is the only truly correct one (even when the variances of the variables differ greatly), and that whenever this version cannot be used, correlation based PCA should not be used either.

I know that there is this thread: PCA on correlation or covariance? -- but it seems to focus only on finding a pragmatic solution, which may or may not also be an algebraically correct one.