Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I was watching a tutorial on topic modeling and no-where they talk if the number in the bag of words model is significant.
i.e. they only care whether word "a" belongs in the document or not, how many times the word "a" appears in the document doesn't matter.
So, how will we apply topic modeling on documents represented as bag of words, wouldn't not using the counts lead to too much information loss?
Are the counts even signifacnt in finding the topics?
Bag of words just means ignoring order---word counts are taken into account and they are important. Maybe the tutorial was just showing a basic example. Generally in LDA documents are represented as word count vectors.
As @conjugateprior says in the comments, the dirichlet distribution depends on these counts. My understanding is that in the generative process for LDA, a distribution over topics is drawn and then, for each topic, words are drawn from a distribution over words. So one word might be drawn multiple times. LDA essentially allows you to infer these distributions for a corpus using the text data. If a word thus appears multiple times in a document, it would lead to a corresponding higher weight in the inferred distribution. Without counts, I think you would just be assuming that each topic has a uniform distribution over words— in LDA though a dirichlet distribution is used. Probably the uniform would work, but as you said there is important information in the count of words.
As mentioned in the previous post, you were probably being taught a simpler version of LSA. Please see the last paragraph of page 3 in
https://web.stanford.edu/~jurafsky/slp3/16.pdf
where the authors describe how frequency and entropy information is generally incorporated into the term document matrix for LSA
