What is VectorSource and VCorpus in 'tm' (Text Mining) package in R

Question

I'm not quite sure what exactly VectorSource and VCorpus are in 'tm' package.

The documentation is unclear on these, can anyone make me understand in simple terms?

Answer 1

"Corpus" is a collection of text documents.

VCorpus in tm refers to "Volatile" corpus which means that the corpus is stored in memory and would be destroyed when the R object containing it is destroyed.

Contrast this with PCorpus or Permanent Corpus which are stored outside the memory in a db.

In order to create a VCorpus using tm, we need to pass a "Source" object as a parameter to the VCorpus method. You can find the sources available using this method -
getSources()

[1] "DataframeSource" "DirSource" "URISource" "VectorSource"
[5] "XMLSource" "ZipSource"

Source abstracts input locations, like a directory, a URI etc. VectorSource is for only character vectors

A simple example :

Say you have a char vector -

input <- c('This is line one.','And this is the second one')

Create the source -

vecSource <- VectorSource(input)

Then create the corpus -

VCorpus(vecSource) 

Hope this helps. You can read more here - https://cran.r-project.org/web/packages/tm/vignettes/tm.pdf

Answer 2

In practical terms, there is a big difference between Corpus and VCorpus.

Corpus uses SimpleCorpus as a default, which means some features of VCorpus will not be available. One that is immediately evident is that SimpleCorpus will not allow you to keep dashes, underscores or other signs of punctuation; SimpleCorpus or Corpus automatically removes them, VCorpus does not. There are other limitations of Corpus that you will find in the help with ?SimpleCorpus.

Here is an example:

# Read a text file from internet
filePath <- "http://www.sthda.com/sthda/RDoc/example-files/martin-luther-king-i-have-a-dream-speech.txt"
text <- readLines(filePath)

# load the data as a corpus
C.mlk <- Corpus(VectorSource(text))
C.mlk
V.mlk <- VCorpus(VectorSource(text))
V.mlk

The output will be:

<<SimpleCorpus>>
Metadata:  corpus specific: 1, document level (indexed): 0
Content:  documents: 46
<<VCorpus>>
Metadata:  corpus specific: 0, document level (indexed): 0
Content:  documents: 46

If you do an inspection of the objects:

# inspect the content of the document
inspect(C.mlk[1:2])
inspect(V.mlk[1:2])

You will notice that Corpus unpacks the text:

<<SimpleCorpus>>
Metadata:  corpus specific: 1, document level (indexed): 0
Content:  documents: 2
[1]                                                                                                                                            
[2] And so even though we face the difficulties of today and tomorrow, I still have a dream. It is a dream deeply rooted in the American dream.


<<VCorpus>>
Metadata:  corpus specific: 0, document level (indexed): 0
Content:  documents: 2
[[1]]
<<PlainTextDocument>>
Metadata:  7
Content:  chars: 0
[[2]]
<<PlainTextDocument>>
Metadata:  7
Content:  chars: 139

While VCorpus keeps it together within the object.

Let's say now you do the matrix conversion for both:

dtm.C.mlk <- DocumentTermMatrix(C.mlk)
length(dtm.C.mlk$dimnames$Terms)
# 168

dtm.V.mlk <- DocumentTermMatrix(V.mlk)
length(dtm.V.mlk$dimnames$Terms)
# 187

Finally, let's see the content. This is from Corpus:

grep("[[:punct:]]", dtm.C.mlk$dimnames$Terms, value = TRUE)
# character(0)

And from VCorpus:

grep("[[:punct:]]", dtm.V.mlk$dimnames$Terms, value = TRUE)

[1] "alabama,"       "almighty,"      "brotherhood."   "brothers."     
 [5] "california."    "catholics,"     "character."     "children,"     
 [9] "city,"          "colorado."      "creed:"         "day,"          
[13] "day."           "died,"          "dream."         "equal."        
[17] "exalted,"       "faith,"         "gentiles,"      "georgia,"      
[21] "georgia."       "hamlet,"        "hampshire."     "happens,"      
[25] "hope,"          "hope."          "injustice,"     "justice."      
[29] "last!"          "liberty,"       "low,"           "meaning:"      
[33] "men,"           "mississippi,"   "mississippi."   "mountainside," 
[37] "nation,"        "nullification," "oppression,"    "pennsylvania." 
[41] "plain,"         "pride,"         "racists,"       "ring!"         
[45] "ring,"          "ring."          "self-evident,"  "sing."         
[49] "snow-capped"    "spiritual:"     "straight;"      "tennessee."    
[53] "thee,"          "today!"         "together,"      "together."     
[57] "tomorrow,"      "true."          "york."

Take a look at the words with punctuation. That is a huge difference. Isn't it?