3
$\begingroup$

I've generated 100 random vectors (data points) in n∈[1,...,50] dimensions. I then compared distances between each pair of vectors and calculated the mean value. I've done this for all dimensions using Euclidean distance and then using cosine similarity.

Also worth noting is that all vectors are in range [0, 1)

This is the graph I get:

enter image description here

From what I've read on the topic, there shouldn't be a significant difference between these two measurements and both of them will be affected by the curse of the dimensionality.

AFAIK cosine similarity is only in the interval from [-1, 1] and in my case (all vectors are positive) the interval will be [0, 1]. Is that why the cosine similarity is "much" smaller than Euclidean distance? And also, how come the cosine similarity curve isn't rising with the number of dimensions?

Note: This is a question in my lab exercise: Which one of these measures would you use in high dimension space?

In my opinion there isn't a clear answer to this.

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ By generating vectors of arbitrary length, you can make the average Euclidean distance become as large or small as you wish. Everything depends, then, on how you generate these random numbers. But that begs the question: because the magnitude of the Euclidean distance is perfectly arbitrary, it makes no sense in general to compare it to the cosine distance. As far as the lab question goes, perhaps the proper response ought to be "for what purpose?" Can you provide some context for that? $\endgroup$
    – whuber ♦
    Commented Dec 16, 2021 at 20:00
  • 1
    $\begingroup$ Shouldn't the answer depend on the nature of the data and the costs of mis-classification? $\endgroup$
    – whuber ♦
    Commented Dec 16, 2021 at 20:13
  • 1
    $\begingroup$ Thank you for accepting my answer, but Cross Validated etiquette says that you should wait longer than a few minutes after posting a question to accept an answer. People tend not to reply to questions with accepted answers, meaning that an even better answer than mine is unlikely. $\endgroup$
    – Dave
    Commented Dec 16, 2021 at 20:18
  • 2
    $\begingroup$ Do not forget that cosine is based on vectors of normalized, unit length. CS = 1 - (d^2)/2, where d is the chord distance (a particular case of euclidean distance). $\endgroup$
    – ttnphns
    Commented Dec 16, 2021 at 20:19
  • 1
    $\begingroup$ My comment contained a hint. Recall that as you add dimensions, the bulk volume of points more and more crawls toward the corners of the hypercube. But the hypersphere - which is what about chord/cosine measure keeps points in the central region. $\endgroup$
    – ttnphns
    Commented Dec 16, 2021 at 20:30

1 Answer 1

Reset to default
4
$\begingroup$

This makes sense to me. When you add more dimensions, you are, in some sense, giving more ways for the points to be far apart. On the number line, $x_0=0$ and $x_1 = 1$ are only $1$ unit apart. However, if we add a second dimension, $(x_0, y_0) = (0, 0)$ and $(x_1, y_1) = (1, 100)$ are a lot more than one unit apart. However, the angular measurement is always restricted to just the two dimensions spanned by your two points.

We have two questions on here that might interest you.

Why is Euclidean distance not a good metric in high dimensions?

Square loss for "big data"

EDIT

You can decide if this makes you like or dislike cosine distance, but consider the points $(0, 1)\in\mathbb R^2$ and $(1, 0)\in\mathbb R^2$. They have the same cosine distance as $(0, 1)$ and $(2, 0)$, but the Euclidean distances are different.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I don't follow the logic. If you want information useful for distinguishing data, why wouldn't the Euclidean distance perforce be preferable to a bounded metric like the cosine distance? ;-) $\endgroup$
    – whuber ♦
    Commented Dec 16, 2021 at 20:14
  • $\begingroup$ Regarding which you should use in high dimensions, without more context, the question is too open-ended, though my guess is that the full-credit answer to your homework involves arguing in favor of cosine similarity. $\endgroup$
    – Dave
    Commented Dec 16, 2021 at 20:14
  • $\begingroup$ This makes intuitive sense to me, but this answer made me think otherwise so I wasn't sure. $\endgroup$
    – Jamess11
    Commented Dec 16, 2021 at 20:15
  • $\begingroup$ @whuber I do not see where I argue in favor of cosine distance. Could you please point me to where you see it? I would like to edit the post to clarify. $\endgroup$
    – Dave
    Commented Dec 16, 2021 at 20:16
  • 1
    $\begingroup$ You have more space to work with to be far apart. The Euclidean distance is going to increase unless points have exactly the same value of the extra vector components. $\endgroup$
    – Dave
    Commented Dec 16, 2021 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.