Skip to main content
deleted 9 characters in body
Source Link

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.

I have a query vector and some texts, all vectors are constructed using the TF-IDF algorithm usingand the same corpus with the same preprocessing, including Porter stemmer. The problem is that no matter the texts andnor the query vector, both cosine and euclidean distances selectchose the same text. For instance, let $u$ be the query vector and $v_1, v_2, ..., v_n$ be the vectorized texts, then cosine similarity gives selects the vector $v_k$ to be more similar to $u$, and so euclidean similarity measure. So, it's weird because I'm getting different results of similarity for those vectors, but they still chose the same vector. So my questions are:

  • Cosine and euclidean similarities select the same vectors always?
  • Is mandatory to normalize the euclidean distance?
  • Why cosine distance shouldn't be normalized?

Notes:

  • To calculate the similarity coefficient of euclidean distance I use the formula ${1}\over{1+euclidean\_distance(u,v)}$ and for cosine distance $1-cosine\_distance(u,v)$.
  • Note that normalized means that the sum of all components of a vector is 1.

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.

I have a query vector and some texts, all vectors are constructed using the TF-IDF algorithm using the same corpus with the same preprocessing, including Porter stemmer. The problem is that no matter the texts and the query vector, both cosine and euclidean distances select the same text. For instance, let $u$ be the query vector and $v_1, v_2, ..., v_n$ be the vectorized texts, then cosine similarity gives selects the vector $v_k$ to be more similar to $u$, and so euclidean similarity measure. So, it's weird because I'm getting different results of similarity for those vectors, but they still chose the same vector. So my questions are:

  • Cosine and euclidean similarities select the same vectors always?
  • Is mandatory to normalize the euclidean distance?
  • Why cosine distance shouldn't be normalized?

Notes:

  • To calculate the similarity coefficient of euclidean distance I use the formula ${1}\over{1+euclidean\_distance(u,v)}$ and for cosine distance $1-cosine\_distance(u,v)$.
  • Note that normalized means that the sum of all components of a vector is 1.

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.

I have a query vector and some texts, all vectors are constructed using the TF-IDF algorithm and the same corpus with the same preprocessing, including Porter stemmer. The problem is that no matter the texts nor the query vector, both cosine and euclidean distances chose the same text. For instance, let $u$ be the query vector and $v_1, v_2, ..., v_n$ be the vectorized texts, then cosine similarity selects the vector $v_k$ to be more similar to $u$, and so euclidean similarity measure. So, it's weird because I'm getting different results of similarity for those vectors, but they still chose the same vector. So my questions are:

  • Cosine and euclidean similarities select the same vectors always?
  • Is mandatory to normalize the euclidean distance?
  • Why cosine distance shouldn't be normalized?

Notes:

  • To calculate the similarity coefficient of euclidean distance I use the formula ${1}\over{1+euclidean\_distance(u,v)}$ and for cosine distance $1-cosine\_distance(u,v)$.
  • Note that normalized means that the sum of all components of a vector is 1.
Source Link

Similarity index between two texts Ask Question

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.

I have a query vector and some texts, all vectors are constructed using the TF-IDF algorithm using the same corpus with the same preprocessing, including Porter stemmer. The problem is that no matter the texts and the query vector, both cosine and euclidean distances select the same text. For instance, let $u$ be the query vector and $v_1, v_2, ..., v_n$ be the vectorized texts, then cosine similarity gives selects the vector $v_k$ to be more similar to $u$, and so euclidean similarity measure. So, it's weird because I'm getting different results of similarity for those vectors, but they still chose the same vector. So my questions are:

  • Cosine and euclidean similarities select the same vectors always?
  • Is mandatory to normalize the euclidean distance?
  • Why cosine distance shouldn't be normalized?

Notes:

  • To calculate the similarity coefficient of euclidean distance I use the formula ${1}\over{1+euclidean\_distance(u,v)}$ and for cosine distance $1-cosine\_distance(u,v)$.
  • Note that normalized means that the sum of all components of a vector is 1.