37
$\begingroup$

I am trying to understand what is similarity between Latent Dirichlet Allocation and word2vec for calculating word similarity.

As I understand, LDA maps words to a vector of probabilities of latent topics, while word2vec maps them to a vector of real numbers (related to singular value decomposition of pointwise mutual information, see O. Levy, Y. Goldberg, "Neural Word Embedding as Implicit Matrix Factorization"; see also How does word2vec work?).

I am interested both in theoretical relations (can one be considered a generalization, or variation of the other) and practical (when to use one but not the other).

Related:

$\endgroup$
  • $\begingroup$ I found this presentation to be on-spot: slideshare.net/ChristopherMoody3/… $\endgroup$ – Piotr Migdal Jan 20 '16 at 8:52
  • $\begingroup$ You ought to look at Doc2vec (aka. paragraph2vec). Document vectors summarises the document instead of words. $\endgroup$ – sachinruk Dec 29 '16 at 3:44
17
$\begingroup$

An answer to Topic models and word co-occurrence methods covers the difference (skip-gram word2vec is compression of pointwise mutual information (PMI)).

So:

  • neither method is a generalization of another,
  • word2vec allows us to use vector geometry (like word analogy, e.g. $v_{king} - v_{man} + v_{woman} \approx v_{queen}$, I wrote an overview of word2vec)
  • LDA sees higher correlations than two-element,
  • LDA gives interpretable topics.

Some difference is discussed in the slides word2vec, LDA, and introducing a new hybrid algorithm: lda2vec - Christopher Moody.

$\endgroup$
  • $\begingroup$ I'd caveat the statement "LDA gives interpretable topics" to say that LDA's topics are potentially interpretable. LDA's idea of "topic" is a purely mathematical construct that doesn't always map into what a human thinks of as a topic. $\endgroup$ – Wayne Feb 20 at 12:18
  • $\begingroup$ A key concept you left out is that LDA uses a bag-of-words approach, so it only knows about co-occurrences within a document, while word2vec (or more comparably doc2vec) considers a word's context. $\endgroup$ – Wayne Feb 20 at 12:19
12
$\begingroup$

The two algorithms differ quite a bit in their purpose.

LDA is aimed mostly at describing documents and document collections by assigning topic distributions to them, which in turn have word distributions assigned, as you mention.

word2vec looks to embed words in a latent factor vector space, an idea originating from the distributed representations of Bengio et al. It can also be used to describe documents, but is not really designed for the task.

$\endgroup$
  • 1
    $\begingroup$ You could theoretically get something analogous to word2vec's vector embeddings by computing P(topic | word) from LDA, but as @Bar said these models were designed for different tasks. If you compared LDA's P(topic | word) distributions with word2vec's vector embeddings, I doubt they would be very similar. LDA is capturing document level associations while word2vec is capturing very local ones. $\endgroup$ – Zubin Aug 15 '15 at 15:44
2
$\begingroup$

There is a relation between LDA and $\bf {Topic2Vec}$, a model used for learning Distributed Topic Representations $\bf together\ with$ Word Representations. LDA is used to construct a log-likelihood for CBOW and Skip-gram. The following explanation is inside the section 3 of the work Topic2Vec: Learning Distributed Representations of Topics:

"When training, given a word-topic sequence of a document $D=\{w_1 : z_1, ...,w_M : z_M \}$, where $z_i$ is the word $w_i$'s topic inferred from LDA, the learning objective functions can be defined to maximize the following log-likelihoods, based on CBOW and Skip-gram, respectively."

$$\mathcal{L}_{CBOW}(D) = \frac1M \sum^{M}_{i=1}(\log p(w_i|w_{ext}) + \log p(z_i|w_{ext}))$$

$$\mathcal{L}_{Skip-gram}(D)= \frac1M \sum^{M}_{i=1}\sum_{-k\le c\le k,c\neq0}(\log p(w_{i+c}|w_i) + \log p(w_{i+c}|z_i))$$

In section 4.2, the authors explain: " topics and words are equally represented as the low-dimensional vectors, we can IMMEDIATELY CALCULATE THE $\bf {COSINE\ SIMILARITY}$ between words and topics. For each topic, we select higher similarity words".

Moreover, you wil find inside that work some phrases like:

"probability is not the best choice for feature representation"

and

"LDA prefers to describe the statistical relationship of occurrences rather than real semantic information embedded in words, topics and documents"

which will help you understanding better the different models.

$\endgroup$
1
$\begingroup$

Other answers here cover the technical differences between those two algorithms, however I think the core difference is their purpose: Those two algorithms were designed to do different things:

word2vec ultimately yields a mapping between words and a fixed length vector. If we were to compare it with another well known approach, it would make more sense to do so using another tool that was designed for the same intend, like the Bag of Words (BOW model). This one does the same but lacks some desired features of word2vec like using the order of words and assigning semantic meaning to the distances between word representations.

LDA on the other hand creates a mapping from a varied length document to a vector. This document can be a sentence, paragraph or full text file but it is not a single word. It would make more sense to compare it with doc2vec that does the same job and is introduced by Tomas Mikolov here (the author uses the term paragraph vectors). Or with LSI for that matter.

So to directly answer your two questions:

  1. None of them is a generalization or variation of the other
  2. Use LDA to map a document to a fixed length vector. You can then use this vector in a traditional ML algorithm like a classifier that accepts a document and predicts a sentimental label for example.
  3. Use word2vec to map a word to a fixed length vector. You can similarly use these vectors to feed ML models were the input are words, for example when developing an auto-completer that feeds on previous words and attempts to predict the next one.
$\endgroup$
0
$\begingroup$

From a practical standpoint...

LDA starts with a bag-of-words input which considers what words co-occur in documents, but does not pay attention to the immediate context of words. This means the words can appear anywhere in the document and in any order, which strips out a certain level of information. By contrast word2vec is all about the context in which a word is used -- though perhaps not exact order.

LDA's "topics" are a mathematical construct and you shouldn't confuse them with actual human topics. You can end up with topics that have no human interpretation -- they're more like artifacts of the process than actual topics -- and you can end up with topics at different levels of abstraction, including topics that basically cover the same human topic. It's a bit like reading tea leaves.

I've found LDA useful to explore data, but not so useful for providing a solution, but your mileage may vary.

Word2vec doesn't create topics directly at all. It projects words into a high-dimensional space based on similar usage, so it can have its own surprises in terms of words that you think of as distinct -- or even opposite -- may be near each other in space.

You can use either to determine if words are "similar". With LDA: do the words have similar weights in the same topics. With word2vec: are they close (by some measure) in the embedding space.

You can use either to determine if documents are similar. With LDA, you would look for a similar mixture of topics, and with word2vec you would do something like adding up the vectors of the words of the document. ("Document" could be a sentence, paragraph, page, or an entire document.) Doc2vec is a modified version of word2vec that allows the direct comparison of documents.

While LDA throws away some contextual information with its bag-of-words approach, it does have topics (or "topics"), which word2vec doesn't have. So it's straightforward to use doc2vec to say, "Show me documents that are similar to this one", while with LDA it's straightforward to say, "Show me documents where topic A is prominent." (Again, knowing that "topic A" emerges from a mathematical process on your documents and you then figure out what human topic(s) it mostly corresponds to.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.