I've been going through a data science course, and just came across Vector Space Modeling for representing text documents, and don't fully understand it. I've looked at a few articles, but they are pretty high level, and I am just a lowly python developer looking to gain knowledge of the data science world. In one of my course assignments, I was tasked with dealing with VPM, and Naive Bayes together, and I read this stack overflow article about Naive Bayes, and it really helped me understand that algorithm. Could someone similarly explain to me how vector space modeling works, or supply some solid articles that explain it pretty well.


The fundamental idea of a vector space model for text is to treat each distinct term as its own dimension. So, lets say you have a document $D$, of length $M$ words, so we say $w_i$ is the $i$th word in $D$, where $i\in [1...M]$. Furthermore, the set of words contained in $w_i$ form a set called the vocabulary or, more evocatively, the term space, often denoted $\mathcal{V}$.

Here's a concrete example:

Let our actual document $D$ be: "He is neither a friend nor is he a foe"

Then $M=10$, and $w_3$="neither". Our term space consists of all distinct terms in $D$: $\mathcal{V}=\{$"He","is","neither","a","friend","nor","foe"$\}$

Now, lets impose an (arbitrary) ordering on $\mathcal{V}$, so that that we form a basis $V$ of terms. In this basis, $v_i$ refers to the $i$th term in the vocabulary (i.e. we convert the Python "set" $\mathcal{V}$ to a Python "sequence" $V$). Think $V$ = list($\mathcal{V}$)


What we have done is define a basis for a vector space. In this example, we have defined a 7-dimensional vector space, where each term $v_i$ represents an orthogonal axis in a coordinate system much like the traditional x,y,z axes (or, more accurately, $\hat{i},\hat{j},\hat{k}$).

With this space, we now have a convenient way of describing documents: Each document can be represented as a 7-dimensional vector $(n_1,...,n_7)$, where $n_i$ is the number of times term $v_i$ occurs in $D$ (also called the "term frequency"). In our example, we would represent $D$ by projecting it onto our basis $V$, resulting in the following vector:

$$D_{||B}:= (2,2,1,2,1,1,1)$$

This representation forms the core of most text mining methods. For example, you can measure similarity between two documents as the cosine of the angle between their associated vectors. There are many more uses of this method for encoding documents (e.g., see TF-IDF as a refinement of the basic vector space model).

  • 1
    $\begingroup$ Excellent explanation, although it really should be added for newcomers that this is just the "standard" model, known as the "bag-of-words" model, and typically you'd most likely use n-grams or collocations ("bag-of-ngrams", "bag-of-collocations"), or dense representations (e.g., neural embeddings). $\endgroup$ – fnl Jun 30 '17 at 12:41
  • $\begingroup$ @fnl thanks for the addition! Yes, this is the simplest vector space model, one can extend the vocabulary/term space to include n-grams and other objects. Also, there are altrnatives like word2vec....text mining is such a rich area for modeling! Thanks again. $\endgroup$ – user145807 Jun 30 '17 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.