Should we apply PCA before calculating similarities in high-dimensional space if my observations have length 1?

Question

I have high-dimensional space (around 20 features) and I want to calculate similarity based on the angle of observation, not the magnitude. I have a nice function that can compute euclidean distance fast, so I wanted to use it instead of some worse performant function to compute cosine similarities. But as there's direct relationship between euclidean distance and cosine similarity as explained here I think I'm fine as long as I normalise each observations to have length 1. So I did.

But since it is high-dimensional space I feel I could benefit from using PCA. Before PCA, scaling features is advisable as they have ranges and, therefore, variances. But after scaling features I don't have the observations of length 1 anymore, so euclidean distance is not "equivalent" to cosine similarity anymore.

My understanding is that since performing PCA and choosing n principal components changes our features space I should normalise observations in the new feature space and not the old one, because I am calculating similarities on the new feature space. But it still feels like kind of a chicken-or-egg situation.

Am I right about normalising observations in new feature space obtained from PCA? Or should we normalise the observations before PCA? If so, why?

Answer 1

In short, you are right, you should renormalise the observations in the new feature space (after PCA) before calculating cosine similarity.

I assume based on the link you sent you are using (1)

$$\cos(\theta)=1−\frac{\Vert x−x'\Vert^2}{2}$$

to calculate cosine similarity instead of (2)

$$\cos(\theta)=\frac{x \cdot x'}{\Vert x\Vert \Vert x'\Vert}.$$

Note that (1) only works when $\Vert x\Vert =\Vert x'\Vert =1$ in the space you are actually calculating it in - i.e. in the PCA transformed space. So you require that $\Vert x\Vert =\Vert x'\Vert =1$ after applying PCA. Otherwise you aren't actually calculating the cosine similarity at all. If you are using (2) then you don't actually need to renormalise at all and your issues go away. If you normalised in the observation space (before PCA) then you would find that after PCA transform and selecting n components, all your transformed observations would have length less than 1. Therefore (1) won't work.

Finally as side note, the cosine similarity computed on the raw observations should be approximately equivalent to the cosine similarity computed after PCA transform*. They will be exactly equivalent if you keep all the principal components as this is just rotating your axes in euclidean space.

*assuming that the original observations are have zero mean in all dimensions.