1
$\begingroup$

I was watching a tutorial on topic modeling and no-where they talk if the number in the bag of words model is significant.

i.e. they only care whether word "a" belongs in the document or not, how many times the word "a" appears in the document doesn't matter.

So, how will we apply topic modeling on documents represented as bag of words, wouldn't not using the counts lead to too much information loss?

Are the counts even signifacnt in finding the topics?

$\endgroup$

2 Answers 2

0
$\begingroup$

Bag of words just means ignoring order---word counts are taken into account and they are important. Maybe the tutorial was just showing a basic example. Generally in LDA documents are represented as word count vectors.

As @conjugateprior says in the comments, the dirichlet distribution depends on these counts. My understanding is that in the generative process for LDA, a distribution over topics is drawn and then, for each topic, words are drawn from a distribution over words. So one word might be drawn multiple times. LDA essentially allows you to infer these distributions for a corpus using the text data. If a word thus appears multiple times in a document, it would lead to a corresponding higher weight in the inferred distribution. Without counts, I think you would just be assuming that each topic has a uniform distribution over words— in LDA though a dirichlet distribution is used. Probably the uniform would work, but as you said there is important information in the count of words.

$\endgroup$
2
  • $\begingroup$ thanks, could you give the step where it shows that the word count is used? $\endgroup$ Feb 4, 2018 at 20:05
  • 2
    $\begingroup$ @david_nadal Anywhere you see a Dirichlet posterior it will be have an exponent that is a count of something plus a prior parameter for the prevalence of that something. $\endgroup$ Feb 4, 2018 at 20:24
0
$\begingroup$

As mentioned in the previous post, you were probably being taught a simpler version of LSA. Please see the last paragraph of page 3 in

https://web.stanford.edu/~jurafsky/slp3/16.pdf

where the authors describe how frequency and entropy information is generally incorporated into the term document matrix for LSA

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.