1
$\begingroup$

In text mining books, I generally see cosine similarity used as a way to assess the similarity in documents; however, by transposing a tf-idf matrix, one can also calculate cosine similarity between words.

I haven't been able to find an authoritative resource on whether or not looking at the cosine similarity between words is valid. It seems like it would be to me, but I don't understand it enough to feel comfortable with it. It appears as though word2vec does look at cosine similarity at a certain point in their algorithm, however, as discussed elsewhere on CV: Why word2vec maximizes the cosine similarity between semantically similar words

When would it be useful to use this as a similarity metric between two words? Given that it is the cosine of the angle between two p-dimensional vectors (where p can be quite large), I'm finding it somewhat abstract.

See Python and R code below that demonstrates the transposing I am talking about to get at cosine similarity between documents or words, using the same tf-idf matrix.

Note that the Python and R outputs below disagree; I believe this is because scikit-learn and tidytext use different implementations of tf-df.


from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

documents = (
  "the sky is blue",
  "the sun is bright",
  "the sun in the sky is bright",
  "we can see the shining sun, the bright sun"
)

tfidf_vectorizer = TfidfVectorizer()
tfidf_matrix = tfidf_vectorizer.fit_transform(documents)
print(tfidf_matrix.shape)
print(tfidf_matrix.transpose().shape)

cosine_similarity(tfidf_matrix)
cosine_similarity(tfidf_matrix.transpose())

library(tidyverse)
library(tidytext)
dat <- tibble(
  id = paste0("d", 1:4),
  text = c("the sky is blue", "the sun is bright", 
           "the sun in the sky is bright",
           "we can see the shining sun, the bright sun")
)
tfidf_matrix <- dat %>% 
  unnest_tokens(word, text) %>% 
  count(id, word) %>% 
  bind_tf_idf(word, id, n) %>% 
  select(id, word, tf_idf) %>% 
  spread(id, tf_idf, fill = 0)
words <- tfidf_matrix$word
tfidf_matrix <- as.matrix(select(tfidf_matrix, -word))
rownames(tfidf_matrix) <- words

lsa::cosine(tfidf_matrix)
lsa::cosine(t(tfidf_matrix))
$\endgroup$
  • 2
    $\begingroup$ You can use it to calculate similarities between whatever you like, the only question is if the results have any meaningful interpretation, but this would depend highly on context. $\endgroup$ – Tim Nov 30 '18 at 19:40
  • $\begingroup$ I suppose what I'm implying is the question: How would you define when it has meaningful interpretation? The cosine between two p-dimensional vectors with a large p is... abstract. I updated the question to include this point. $\endgroup$ – Mark White Nov 30 '18 at 20:50
  • $\begingroup$ Math is abstract. But seriously, what you are saying is that you calculated something for some values and now you seem to be asking why did you calculate it? You tell us why you consider it interesting and meaningful. $\endgroup$ – Tim Nov 30 '18 at 21:37
  • 1
    $\begingroup$ Cosine sim. computed between binary vectors is called also Ochiai coefficient. It could be computed on column vectors or on row vectors. It is you who decides if it makes sense and be interpretable. If two words tend to co-occur in many same texts these words are contextually similar. You should consider also other popular indices often seen as alternatives to Ochial, including Jaccard and Kulzcynski-2. $\endgroup$ – ttnphns Dec 1 '18 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.