Does anyone have experience reducing the dimensions in a traditional bag of words model?

For example, if you want to train a decision tree on a large set of reviews, the size of the vocabulary would lead to the curse of dimensionality. Would it make sense to run latent dirichlet allocation, then take the ~top10 words in each topic and use that set of words to represent all of the vocabulary?


1 Answer 1


This is not really an answer but rather a comment (don't have the rep yet) - in general, LDA is perfect for dimensionality reduction under the bag-of-words assumption. But you wouldn't really do that the way you suggested (or at least I haven't seen it done like this). Using LDA you would estimate two things - topics (that is distributions over the vocabulary) and the document specific mixing proportions (how much of each topic is there in each of the documents?). You can now use those mixing proportions as a lower dimensional representation of your documents (instead of using full vector of term counts for example). Another solution would be to look into word embeddings such as word2vec.

  • $\begingroup$ yeah i think i'm reaching the limits of LDA. It appears there is no way to handle nonsense comments, as they brute force fit them into a topic. I'm trying to develop a way to toss irrelevant documents that don't align with the generated topics, then reduce the dimensions to these topics, then fit a model. $\endgroup$
    – barker
    Jun 17, 2019 at 15:46
  • $\begingroup$ yeah, I had a similar problem with irrelevant document, but if you experiment with pre-processing and parameters you can usually find some useful and some irrelevant topics (and you and up using just some of them). it can be tricky to find the right combination though $\endgroup$
    – yassem
    Jun 17, 2019 at 17:59
  • $\begingroup$ To be clear - I see the advantage of being able to model the inter-topic dependencies better. It's just that you don't really always need that and I'm surprised this is not a more common approach. $\endgroup$
    – yassem
    Jun 17, 2019 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.