I was hoping that someone could simply validate or correct my interpretation of Principal Components Analysis. There are a lot of questions on this site about Principal Components analysis--some listed below. But none of these expressed PCA in the way I refer to below, so I wanted to check if my interpretation was correct or incorrect.
I understand how principal components works as a linear dimensional reduction method, using the spectral decomposition of some matrix $X$. It was wondering if it is appropriate to interpret PCA as simply a change of basis, as we might do in some other situations in linear algebra? So PCA simply takes points expressed in the standard basis and transforms them into points expressed in an eigenvector basis. In this process of transformation, some dimensions with low variance are discarded and hence the resulting dimensional reduction.
Of course the process of finding the eigenvector basis uses numerical optimization, so the resulting projection of the original points into the eigenvector basis will produce some "wiggle" in the points, but otherwise the distances between the points are left unchanged. I am thinking in terms of purely classical PCA here, and not some of the PCA variants where regularization imposes some delibrate properties on the resulting reduced rank projection matrix.