I have texts for a bunch of objects. From each text, I removed the stop words, and took each word as an attribute of the object. I then gave each word a rating based on sentiment analysis, so that the more positive the sentiment is the higher rating they receive.

I now want to compare the objects, using Euclidean Distance. A problem is that the length for the texts vary a lot, from a few sentences to a large amount. So, when I compare an object to a set of them, the ones that I think that should be closer end up further away, because they have way more data, so their ratings are way larger and the distances are too.

Is there a way to standardize the data among objects in the same set, considering objects with few data and objects with large amounts of it?


A much better similarity measure for sparse and high dimensional data is that of cosine similarity: $$ s(x, y) = {x^Ty \over ||x|| ||y||}\\ = {\sum_i x_i y_i \over \sqrt{\sum_i x_i^2} \sqrt{\sum_i y_i^2}} $$ where $x$ and $y$ are constructed from their indivudal components, i.e. $x = (x_1, x_2, \dots, x_N$ for N dimensions. Each $x_i$ is what I call a component.

Some observations:

  • if only one of the components is non-zero, the contribution to the distance will be $0$,
  • $s(x, y) \in [-1, 1]$,
  • $s(x, y) = 0$ if $x$ and $y$ are algebraically independent.
  • $s(x, ax) = 0$ if $a > 0$.

In contrast to the Euclidean distance $||x - y||$:

  • even if one of the components is zero, the contribution to the loss is unbounded,
  • more general, $||x - y|| \in [0, \infty)$,
  • $||x - ax|| > 0$ if $a > 0$.

Consequently, the cosine similarity cannot be dominated by single dimensions. Also, dimensions in which one of the vectors is zero have a rather "neutral" effect on the similarity. This can be quite beneficial in high dimensional, sparse vector spaces.

  • $\begingroup$ Please expand your statements if only one of the components..., even one of the components...; and explain what is "component"? The thing is that what you say in the last paragraph doesn't seem to flow out of what you list above it transparently enough. $\endgroup$ – ttnphns Sep 30 '14 at 19:18
  • $\begingroup$ I use components and dimensions interchangeably. I.e. $x_i$ is the i'th component/dimension of the vector $x$. (I added that to the answer.) $\endgroup$ – bayerj Sep 30 '14 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.