Bag of words vs vector space model?

Question

What is/are the difference/s between these text representation models: Bag of words and vector space model?

Answer 1

Bag-of-words and vector space model refer to different aspects of characterizing a body of text such as a document. They are described well in the textbook "Speech and Language Processing" by Jurafsky and Martin, 2009, in section 23.1 on information retrieval. A more terse reference is "Introduction to Information Retrieval" by Manning, Raghavan, and Schütze, 2008, in the section on "The vector space model for scoring".

Bag-of-words refers to what kind of information you can extract from a document (namely, unigram words). Vector space model refers to the data structure for each document (namely, a feature vector of term & term weight pairs). Both aspects complement each other.

More specifically:

Bag-of-words: For a given document, you extract only the unigram words (aka terms) to create an unordered list of words. No POS tag, no syntax, no semantics, no position, no bigrams, no trigrams. Only the unigram words themselves, making for a bunch of words to represent the document. Thus: Bag-of-words.

Vector space model: Given the bag of words that you extracted from the document, you create a feature vector for the document, where each feature is a word (term) and the feature's value is a term weight. The term weight might be:

• a binary value (with 1 indicating that the term occurred in the document, and 0 indicating that it did not);
• a term frequency value (indicating how many times the term occurred in the document); or
• a TF-IDF value (e.g. a small floating-point number like 1.23).

The entire document is thus a feature vector, and each feature vector corresponds to a point in a vector space. The model for this vector space is such that there is an axis for every term in the vocabulary, and so the vector space is V-dimensional, where V is the size of the vocabulary. The vector should then conceptually also be V-dimensional with a feature for every vocabulary term. However, because the vocabulary can be large (on the order of V=100,000s of terms), a document's feature vector typically will contain only the terms that occurred in that document and omit the terms that did not. Such a feature vector is considered sparse.

An example vector representation of a document thus might look like this:

DOCUMENT_ID_42 LABEL_POLITICS a 55 ability 1 about 5 absent 2 abuse 1 access 1
accompanied 1 accompanying 2 according 2 account 1 accounted 1 accurate 1
acknowledge 4 activities 1 actual 1 actually 2 administering 1 ...

where this example vector has a document id (e.g. 42), a ground-truth label (e.g. politics) and a list of features and feature values comprising term & term frequency pairs. Here, it can be seen that the word "absent" occurred 2 times in this document.

Answer 2

Is it that using Bag of Words you assign word frequency to document-term matrix element and in Vector Space Model document-term matrix elements are quite general as long as operations (dot product) in vector space make sense (tf-idf weights, for example)?