0
$\begingroup$

Does a transformation exist that allows to use of the Euclidean distance with the word embeddings? The Cosine distance could be a problem in my case. For example, what if I translate the vector to a polar coordinate system from the cartesian one? Does it make sense?

The idea is reduce the dimensionality of such vector using PCA or Autoencoders. In my understanding, using PCA on word embeddings means to produce random data.

How can I reduce the dimensionality of the word-embeddings saving the meaning the have?

Improve this question
$\endgroup$
4
  • $\begingroup$ Regardless of coordinate system, distance is the same. Coordinate systems are just a representation of the same space. $\endgroup$
    – Arya McCarthy
    Commented Jun 30, 2023 at 0:54
  • $\begingroup$ Yes, I agree. However, since we care about the direction rather than the module, reducing the vectors to be unitary does not imply information loss. At this point, only the angles matter. Further, yesterday I got a hint about that, which confirmed that in the case of unitary vectors, Euclidean distance matches the cosine distance. The first step is solved. About the meaningfulness of a dimensional reduced version of the embeddings... I'm afraid that it is not working. $\endgroup$
    – ozw1z5rd
    Commented Jun 30, 2023 at 13:54
  • $\begingroup$ Both distance metrics are roughly the same stats.stackexchange.com/questions/544951/… $\endgroup$
    – Tim
    Commented Jun 30, 2023 at 14:56
  • $\begingroup$ @Tim, yes I agree. I had an intuition and the link you are referring is a consolidated known fact. $\endgroup$
    – ozw1z5rd
    Commented Jun 30, 2023 at 16:32

1 Answer 1

Reset to default
1
$\begingroup$

As you mention, a common way to measure similarity between word embeddings is the cosine distance. Why ist that? Its because word embeddings are high dimensional, 100-1000 dimensions are common. In this dimensionality everything is sparse and distance are always huge, a "phenomenon" that is called "curse of dimensionality". Cosine distance only takes the angle into account.

You could project your datapoints onto a sphere (should be the same as normalizing the length to 1) and then run a PCA for dimensionality reduction. Now euclidean distances are more comparable to those of their cosine counterparts in the original embedding.

edit I missed OP's comment about this approach not working. Why is that? Don't expect those distances to be identical or similar in magnitude, rather they relation should be somewhat constant. That is, distances are scaled between both "worlds", but should have a constant relation.

Improve this answer
$\endgroup$
3
  • $\begingroup$ if the dimensional reduction did not dramatically alter the information... why the clusters are totally different? $\endgroup$
    – ozw1z5rd
    Commented Jun 30, 2023 at 15:22
  • $\begingroup$ As mentioned by user Tim, I guess this approach is the way to go. I guess you have implementation issues? Maybe provide some code? $\endgroup$
    – Klops
    Commented Jun 30, 2023 at 17:05
  • $\begingroup$ let me say that the lost of variance introduced in the dimensional reduction is not welcomed, so, I cluster the data BEFORE, extract the cluster number and then attach this value as a categorical feature to the final data_set for further (soft/hierarchical/fuzzy) clustering. Quite happy about this solution. From 8,5K feature vectors -> less than 1K feature vectors. $\endgroup$
    – ozw1z5rd
    Commented Jul 3, 2023 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.