0
$\begingroup$

I installed the tm library and want to build n-grams of a corpus using the NGramTokenizer from the RWeka library.

The tm paper p. 11 lists the following call:

> TermDocMatrix(col, control=list(tokenize = NGramTokenizer))

I know I need to use TermDocumentMatrix in the version of tm I'm using. I get the following return:

> col
A corpus with 3 text documents
>
> TermDocumentMatrix(col, control=list(tokenize=NGramTokenizer))
#  fatal error: mach_msg (send) failed: 0x10000003

#  fatal error: mach_msg (send) failed: 0x10000003

A term-document matrix (0 terms, 3 documents)

Non-/sparse entries: 0/0
Sparsity           : NaN%
Maximal term length: 0 
Weighting          : term frequency (tf)
Warnmeldung:
In is.na(x) :
  is.na() auf nicht-(Liste oder Vektor) des Typs 'NULL' angewendet

What could be the problem here? The corpus is not that large (truncated for testing).

> print(object.size("col"))
96 bytes
$\endgroup$
3
  • 1
    $\begingroup$ I'd be happy to answer by explaining how to do this in the quanteda package (see stackoverflow.com/questions/31281534/…), but you did ask specifically how to do it in tm. Let me know if you would find such an answer helpful. $\endgroup$
    – Ken Benoit
    Jul 12, 2015 at 0:20
  • $\begingroup$ I would find it useful. $\endgroup$
    – TMOTTM
    Jul 13, 2015 at 17:34
  • $\begingroup$ OK done, see my answer below. $\endgroup$
    – Ken Benoit
    Jul 14, 2015 at 1:36

2 Answers 2

8
$\begingroup$

I cannot quickly see what would generate your error without having a sample data set. But if you run the following code you can see that the RWeka tokonizer works.

  library(tm)
  library(RWeka)

  data("crude")
  crude <- as.VCorpus(crude)
  crude <- tm_map(crude, stripWhitespace)
  crude <- tm_map(crude, content_transformer(tolower))
  crude <- tm_map(crude, removeWords, stopwords("english"))
  crude <- tm_map(crude, stemDocument)
  # Sets the default number of threads to use
  options(mc.cores=1)

  tdm <- TermDocumentMatrix(crude, control=list(tokenize = NGramTokenizer))

  findFreqTerms(tdm, lowfreq = 10)

[1] "accord"      "barrel"      "bpd"         "crude"       "crude oil"   "dlrs"        "dlrs barrel" "futur"       "govern"      "kuwait"      "last"       
[12] "market"      "meet"        "mln"         "mln bpd"     "month"       "new"         "offici"      "oil"         "oil price"   "one"         "opec"       
[23] "pct"         "price"       "prices"      "product"     "reuter"      "said"        "saudi"       "say"         "sheikh"      "u s"         "will"       
[34] "world"       "year"       
$\endgroup$
2
  • $\begingroup$ I tried to reproduce your example: > crude <- tm_map(crude, stripWhitespace) > crude <- tm_map(crude, content_transformer(tolower)) Warnmeldung: In parallel::mclapply(x, FUN, ...) : all scheduled cores encountered errors in user code $\endgroup$
    – TMOTTM
    Jul 13, 2015 at 17:32
  • $\begingroup$ Ah, I have seen that mentioned before. RWeka and parallel apparently have some errors working together. See also this post on SO: stackoverflow.com/questions/17703553/…. I have adjusted a line of code that should help. $\endgroup$
    – phiver
    Jul 14, 2015 at 6:50
7
$\begingroup$

OK, here is the method for tokenizing grams in quanteda. Our view is that there is no such thing as n-grams without tokenization, since the notion implies sequences of tokens defined by some kind of adjacency. So we built in an ngrams option into our tokenize() function. ngrams takes a vector of integers, where each integer represents a size of ngram (the default is 1).

It works like this:

> # or: devtools::install_github("kbenoit/quanteda")
> require(quanteda)
> mytext <- c("The quick brown fox, jumped over the lazy dog.",
+             "Here is a second sentence to tokenize.")
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 2)
[[1]]
[1] "the_quick"   "quick_brown" "brown_fox"   "fox_jumped"  "jumped_over" "over_the"    "the_lazy"   
[8] "lazy_dog"   

[[2]]
[1] "here_is"         "is_a"            "a_second"        "second_sentence" "sentence_to"     "to_tokenize"    

attr(,"class")
[1] "tokenizedTexts" "list"          
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 3)
[[1]]
[1] "the_quick_brown"  "quick_brown_fox"  "brown_fox_jumped" "fox_jumped_over"  "jumped_over_the" 
[6] "over_the_lazy"    "the_lazy_dog"    

[[2]]
[1] "here_is_a"            "is_a_second"          "a_second_sentence"    "second_sentence_to"  
[5] "sentence_to_tokenize"

attr(,"class")
[1] "tokenizedTexts" "list"          
> tokenize(toLower(mytext), removePunct = TRUE, ngrams = 1:3)
[[1]]
 [1] "the"              "quick"            "brown"            "fox"              "jumped"          
 [6] "over"             "the"              "lazy"             "dog"              "the_quick"       
[11] "quick_brown"      "brown_fox"        "fox_jumped"       "jumped_over"      "over_the"        
[16] "the_lazy"         "lazy_dog"         "the_quick_brown"  "quick_brown_fox"  "brown_fox_jumped"
[21] "fox_jumped_over"  "jumped_over_the"  "over_the_lazy"    "the_lazy_dog"    

[[2]]
 [1] "here"                 "is"                   "a"                    "second"              
 [5] "sentence"             "to"                   "tokenize"             "here_is"             
 [9] "is_a"                 "a_second"             "second_sentence"      "sentence_to"         
[13] "to_tokenize"          "here_is_a"            "is_a_second"          "a_second_sentence"   
[17] "second_sentence_to"   "sentence_to_tokenize"

attr(,"class")
[1] "tokenizedTexts" "list"          
> 
> # to create a dfm from one of these
> mytokTxts <- tokenize(toLower(mytext), removePunct = TRUE, ngrams = 3)
> dfm(mytokTxts)
Creating a dfm from a tokenizedTexts object ...
   ... indexing 2 documents
   ... shaping tokens into data.table, found 12 total tokens
   ... summing tokens by document
   ... indexing 12 feature types
   ... building sparse matrix
   ... created a 2 x 12 sparse dfm
   ... complete. Elapsed time: 0.023 seconds.
Document-feature matrix of: 2 documents, 12 features.
2 x 12 sparse Matrix of class "dfmSparse"
       features
docs    a_second_sentence brown_fox_jumped fox_jumped_over here_is_a is_a_second jumped_over_the over_the_lazy
  text1                 0                1               1         0           0               1             1
  text2                 1                0               0         1           1               0             0
       features
docs    quick_brown_fox second_sentence_to sentence_to_tokenize the_lazy_dog the_quick_brown
  text1               1                  0                    0            1               1
  text2               0                  1                    1            0               0

Happy to help further if you have more questions.

UPDATED 2021 w/quanteda v3

Here's how to do it with a less ancient version of quanteda:

library("quanteda")
## Package version: 3.0.
## Unicode version: 13.0
## ICU version: 69.1
## Parallel computing: 12 of 12 threads used.
## See https://quanteda.io for tutorials and examples.

mytext <- c(
  "The quick brown fox, jumped over the lazy dog.",
  "Here is a second sentence to tokenize."
)

tokens(mytext, remove_punct = TRUE) %>%
  tokens_tolower() %>%
  tokens_ngrams(n = 2) %>%
  dfm()
## Document-feature matrix of: 2 documents, 14 features (50.00% sparse) and 0 docvars.
##        features
## docs    the_quick quick_brown brown_fox fox_jumped jumped_over over_the
##   text1         1           1         1          1           1        1
##   text2         0           0         0          0           0        0
##        features
## docs    the_lazy lazy_dog here_is is_a
##   text1        1        1       0    0
##   text2        0        0       1    1
## [ reached max_nfeat ... 4 more features ]
```
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.