Similarity index between two texts Ask Question

Question

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.

I have a query vector and some texts, all vectors are constructed using the TF-IDF algorithm and the same corpus with the same preprocessing, including Porter stemmer. The problem is that no matter the texts nor the query vector, both cosine and euclidean distances chose the same text. For instance, let $u$ be the query vector and $v_1, v_2, ..., v_n$ be the vectorized texts, then cosine similarity selects the vector $v_k$ to be more similar to $u$, and so euclidean similarity measure. So, it's weird because I'm getting different results of similarity for those vectors, but they still chose the same vector. So my questions are:

• Cosine and euclidean similarities select the same vectors always?
• Is mandatory to normalize the euclidean distance?
• Why cosine distance shouldn't be normalized?

Notes:

• To calculate the similarity coefficient of euclidean distance I use the formula ${1}\over{1+euclidean\_distance(u,v)}$ and for cosine distance $1-cosine\_distance(u,v)$.
• Note that normalized means that the sum of all components of a vector is 1.

Answer 1

For normalized vectors euclid_dist(x,y) gives the same similarity ordering as 1 - cos(x,y). It will hold true also for your similarity coefficient constructed from euclid_dist.

Depending on paramters of tfidfveectorizer, the vectors it produces are more or less or strictly (norm='l2') normalized.

Therefore, especially if you have small corpus, it may happend that both similarity measures give the same results.

Not sure what scenario you are thinking about when asking about normalizing euclid_dist - ? Actually when you want to find the closest document, you can just find the one with smalled euclid_dist.

Cosine distance is always between 0 and 1, so you don't need to normalize it, neither the vectors you are comparing, as cos_dist returns cosine of the angle between them, which is to reflect 'sentiment' of both vectors, ignoring 'magnitute' of their sentiment.