Euclidean distance with sparse and high dimension data I have texts for a bunch of objects. From each text, I removed the stop words, and took each word as an attribute of the object. I then gave each word a rating based on sentiment analysis, so that the more positive the sentiment is the higher rating they receive.
I now want to compare the objects, using Euclidean Distance. A problem is that the length for the texts vary a lot, from a few sentences to a large amount. So, when I compare an object to a set of them, the ones that I think that should be closer end up further away, because they have way more data, so their ratings are way larger and the distances are too.
Is there a way to standardize the data among objects in the same set, considering objects with few data and objects with large amounts of it?
 A: A much better similarity measure for sparse and high dimensional data is that of cosine similarity:
$$
s(x, y) = {x^Ty \over ||x|| ||y||}\\
  = {\sum_i x_i y_i \over \sqrt{\sum_i x_i^2} \sqrt{\sum_i y_i^2}}
$$
where $x$ and $y$ are constructed from their indivudal components, i.e. $x = (x_1, x_2, \dots, x_N$ for N dimensions. Each $x_i$ is what I call a component.
Some observations:


*

*if only one of the components is non-zero, the contribution to the distance will be $0$,

*$s(x, y) \in [-1, 1]$,

*$s(x, y) = 0$ if $x$ and $y$ are algebraically independent.

*$s(x, ax) = 0$ if $a > 0$.


In contrast to the Euclidean distance $||x - y||$:


*

*even if one of the components is zero, the contribution to the loss is unbounded,

*more general, $||x - y|| \in [0, \infty)$,

*$||x - ax|| > 0$ if $a > 0$. 


Consequently, the cosine similarity cannot be dominated by single dimensions. Also, dimensions in which one of the vectors is zero have a rather "neutral" effect on the similarity. This can be quite beneficial in high dimensional, sparse vector spaces.
