In text mining books, I generally see cosine similarity used as a way to assess the similarity in documents; however, by transposing a tf-idf matrix, one can also calculate cosine similarity between words.
I haven't been able to find an authoritative resource on whether or not looking at the cosine similarity between words is valid. It seems like it would be to me, but I don't understand it enough to feel comfortable with it. It appears as though word2vec does look at cosine similarity at a certain point in their algorithm, however, as discussed elsewhere on CV: Why word2vec maximizes the cosine similarity between semantically similar words
When would it be useful to use this as a similarity metric between two words? Given that it is the cosine of the angle between two p-dimensional vectors (where p can be quite large), I'm finding it somewhat abstract.
See Python and R code below that demonstrates the transposing I am talking about to get at cosine similarity between documents or words, using the same tf-idf matrix.
Note that the Python and R outputs below disagree; I believe this is because
tidytext use different implementations of tf-df.
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity documents = ( "the sky is blue", "the sun is bright", "the sun in the sky is bright", "we can see the shining sun, the bright sun" ) tfidf_vectorizer = TfidfVectorizer() tfidf_matrix = tfidf_vectorizer.fit_transform(documents) print(tfidf_matrix.shape) print(tfidf_matrix.transpose().shape) cosine_similarity(tfidf_matrix) cosine_similarity(tfidf_matrix.transpose())
library(tidyverse) library(tidytext) dat <- tibble( id = paste0("d", 1:4), text = c("the sky is blue", "the sun is bright", "the sun in the sky is bright", "we can see the shining sun, the bright sun") ) tfidf_matrix <- dat %>% unnest_tokens(word, text) %>% count(id, word) %>% bind_tf_idf(word, id, n) %>% select(id, word, tf_idf) %>% spread(id, tf_idf, fill = 0) words <- tfidf_matrix$word tfidf_matrix <- as.matrix(select(tfidf_matrix, -word)) rownames(tfidf_matrix) <- words lsa::cosine(tfidf_matrix) lsa::cosine(t(tfidf_matrix))