I'm working with scRNA-seq data (~96% sparse, high dimensional), and am trying to determine distances between the cells in PCA space - NOT for the specific purpose of clustering. The principal components themselves are not sparse or high dimensional (only 20 PCs), but as the original data are I'm struggling to determine the best distance metric from a theoretical standpoint. I've read that theoretically, if all PCs are retained, that the Euclidean distance in PCA is equivalent to the Euclidean distance in the n dimensional space. With that, and the poor performance of Euclidean distance in sparse, high n space in mind, I think Euclidean distance is out.
Next I considered the cosine distance, which according to my research and especially this answer, isn't much better than Euclidean for sparse, high n data (as their formulae are very similar), but is at least faster, which is a concern since the datasets I'm working with are large. However, I've failed to find a good, fast implementation of cosine dissimilarity calculation for a matrix in R. I wrote one, but it's memory and computationally intensive.
Finally, I decided to consider other, more bespoke methods. I've looked through Can Shared-Neighbor Distances Defeat the Curse of Dimensionality?, which is certainly intriguing, but as the PCA data isn't really high-dimensional, I'm not sure if it's the correct method to use (and I also haven't found any code implementation of the method).
I've also read through Improved sqrt-cosine similarity measurement, which also seems promising. It faces the same issue of compute time / lack of implementation that the normal cosine distance does.
Any thoughts or advice? I've done my best to do my due diligence on methodology, and I've read through many similar questions on this forum, but haven't been able to find what I would consider a satisfactory conclusion, although I would love it if I've simply just missed the answer in my Googling.
