I installed the tm library and want to build n-grams of a corpus using the NGramTokenizer from the RWeka library.

The tm paper p. 11 lists the following call:

> TermDocMatrix(col, control=list(tokenize = NGramTokenizer))

I know I need to use TermDocumentMatrix in the version of tm I'm using. I get the following return:

> col
A corpus with 3 text documents
> TermDocumentMatrix(col, control=list(tokenize=NGramTokenizer))
#  fatal error: mach_msg (send) failed: 0x10000003

#  fatal error: mach_msg (send) failed: 0x10000003

A term-document matrix (0 terms, 3 documents)

Non-/sparse entries: 0/0
Sparsity           : NaN%
Maximal term length: 0 
Weighting          : term frequency (tf)
In is.na(x) :
  is.na() auf nicht-(Liste oder Vektor) des Typs 'NULL' angewendet

What could be the problem here? The corpus is not that large (truncated for testing).

> print(object.size("col"))
96 bytes
  • 1
    $\begingroup$ I'd be happy to answer by explaining how to do this in the quanteda package (see stackoverflow.com/questions/31281534/…), but you did ask specifically how to do it in tm. Let me know if you would find such an answer helpful. $\endgroup$ – Ken Benoit Jul 12 '15 at 0:20
  • $\begingroup$ I would find it useful. $\endgroup$ – TMOTTM Jul 13 '15 at 17:34
  • $\begingroup$ OK done, see my answer below. $\endgroup$ – Ken Benoit Jul 14 '15 at 1:36

I cannot quickly see what would generate your error without having a sample data set. But if you run the following code you can see that the RWeka tokonizer works.


  crude <- as.VCorpus(crude)
  crude <- tm_map(crude, stripWhitespace)
  crude <- tm_map(crude, content_transformer(tolower))
  crude <- tm_map(crude, removeWords, stopwords("english"))
  crude <- tm_map(crude, stemDocument)
  # Sets the default number of threads to use

  tdm <- TermDocumentMatrix(crude, control=list(tokenize = NGramTokenizer))

  findFreqTerms(tdm, lowfreq = 10)

[1] "accord"      "barrel"      "bpd"         "crude"       "crude oil"   "dlrs"        "dlrs barrel" "futur"       "govern"      "kuwait"      "last"       
[12] "market"      "meet"        "mln"         "mln bpd"     "month"       "new"         "offici"      "oil"         "oil price"   "one"         "opec"       
[23] "pct"         "price"       "prices"      "product"     "reuter"      "said"        "saudi"       "say"         "sheikh"      "u s"         "will"       
[34] "world"       "year"       
  • $\begingroup$ I tried to reproduce your example: > crude <- tm_map(crude, stripWhitespace) > crude <- tm_map(crude, content_transformer(tolower)) Warnmeldung: In parallel::mclapply(x, FUN, ...) : all scheduled cores encountered errors in user code $\endgroup$ – TMOTTM Jul 13 '15 at 17:32
  • $\begingroup$ Ah, I have seen that mentioned before. RWeka and parallel apparently have some errors working together. See also this post on SO: stackoverflow.com/questions/17703553/…. I have adjusted a line of code that should help. $\endgroup$ – phiver Jul 14 '15 at 6:50

OK, here is the method for tokenizing grams in quanteda. Our view is that there is no such thing as n-grams without tokenization, since the notion implies sequences of tokens defined by some kind of adjacency. So we built in an ngrams option into our tokenize() function. ngrams takes a vector of integers, where each integer represents a size of ngram (the default is 1).

It works like this:

> # or: devtools::install_github("kbenoit/quanteda")
> require(quanteda)
> mytext <- c("The quick brown fox, jumped over the lazy dog.",
+             "Here is a second sentence to tokenize.")
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 2)
[1] "the_quick"   "quick_brown" "brown_fox"   "fox_jumped"  "jumped_over" "over_the"    "the_lazy"   
[8] "lazy_dog"   

[1] "here_is"         "is_a"            "a_second"        "second_sentence" "sentence_to"     "to_tokenize"    

[1] "tokenizedTexts" "list"          
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 3)
[1] "the_quick_brown"  "quick_brown_fox"  "brown_fox_jumped" "fox_jumped_over"  "jumped_over_the" 
[6] "over_the_lazy"    "the_lazy_dog"    

[1] "here_is_a"            "is_a_second"          "a_second_sentence"    "second_sentence_to"  
[5] "sentence_to_tokenize"

[1] "tokenizedTexts" "list"          
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 1:3)
 [1] "the"              "quick"            "brown"            "fox"              "jumped"          
 [6] "over"             "the"              "lazy"             "dog"              "the_quick"       
[11] "quick_brown"      "brown_fox"        "fox_jumped"       "jumped_over"      "over_the"        
[16] "the_lazy"         "lazy_dog"         "the_quick_brown"  "quick_brown_fox"  "brown_fox_jumped"
[21] "fox_jumped_over"  "jumped_over_the"  "over_the_lazy"    "the_lazy_dog"    

 [1] "here"                 "is"                   "a"                    "second"              
 [5] "sentence"             "to"                   "tokenize"             "here_is"             
 [9] "is_a"                 "a_second"             "second_sentence"      "sentence_to"         
[13] "to_tokenize"          "here_is_a"            "is_a_second"          "a_second_sentence"   
[17] "second_sentence_to"   "sentence_to_tokenize"

[1] "tokenizedTexts" "list"          
> # to create a dfm from one of these
> mytokTxts <- tokenize(toLower(mytext), removePunct = TRUE, ngrams = 3)
> dfm(mytokTxts)
Creating a dfm from a tokenizedTexts object ...
   ... indexing 2 documents
   ... shaping tokens into data.table, found 12 total tokens
   ... summing tokens by document
   ... indexing 12 feature types
   ... building sparse matrix
   ... created a 2 x 12 sparse dfm
   ... complete. Elapsed time: 0.023 seconds.
Document-feature matrix of: 2 documents, 12 features.
2 x 12 sparse Matrix of class "dfmSparse"
docs    a_second_sentence brown_fox_jumped fox_jumped_over here_is_a is_a_second jumped_over_the over_the_lazy
  text1                 0                1               1         0           0               1             1
  text2                 1                0               0         1           1               0             0
docs    quick_brown_fox second_sentence_to sentence_to_tokenize the_lazy_dog the_quick_brown
  text1               1                  0                    0            1               1
  text2               0                  1                    1            0               0

Happy to help further if you have more questions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.