3
$\begingroup$

I want to find the document similarity.

I am using the below R code

read 1000 txt articles from directory data/txt

corpus <-Corpus(DirSource("data/txt"), readerControl = list(blank.lines.skip=TRUE));

some preprocessing

corpus <- tm_map(corpus, removeWords, stopwords("english")) corpus <- tm_map(corpus, stripWhitespace) corpus <- tm_map(corpus, stemDocument, language="english")

creating term matrix with TF-IDF weighting

terms <-DocumentTermMatrix(corpus,control = list(weighting = function(x) weightTfIdf(x, normalize = FALSE)))

or compute cosine distance among documents

dissimilarity(tdm, method = "cosine")

It seems dissimilarity function is not supported by the tm package any more. Is there a way to find the cosine angle between the documents. (Rows are documents, columns are texts. So i want to find the similarity between the rows using cosine angle).

$\endgroup$
3
$\begingroup$

You can try using an alternative package, quanteda, that has a function called similarity(). It's still a bit rough around the edges but works as advertised.

require(quanteda)
# clean the texts and create a document-feature matrix
myDfm <- dfm(inaugCorpus, verbose = FALSE)
# similarity matrix for cosine
similarity(myDfm, docnames(myDfm), margin = "documents", method = "cosine")

The last command produces a list of numeric vectors of cosine similarity where each list element corresponds to a document (supplied by docnames(myDfm)). If you wanted to focus on just a single document, you can name just one.

To load the files you want, you can use (might need adjustment for your example):

myCorpus <- corpus(textfile("data/txt/*txt"))
$\endgroup$
  • $\begingroup$ Does cosine angle calculated based on TF-IDF values? $\endgroup$ – Arun Sep 23 '15 at 7:58
  • $\begingroup$ It does not by default, but you can use weight(myDfm, "tfidf") to change the dfm object to tfodf weights, and then compute the cosine distances on that. $\endgroup$ – Ken Benoit Nov 5 '15 at 15:44
  • $\begingroup$ @KenBenoit what is inaugCorpus data type in your code, I tried to put my Corpus object but gives me error saying ` Error in UseMethod("dfm") : no applicable method for 'dfm' applied to an object of class "c('matrix', 'list')"` $\endgroup$ – alily Oct 28 '16 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.