using latent dirichlet allocation to reduce the number of dimensions in bag of words model?

Question

Does anyone have experience reducing the dimensions in a traditional bag of words model?

For example, if you want to train a decision tree on a large set of reviews, the size of the vocabulary would lead to the curse of dimensionality. Would it make sense to run latent dirichlet allocation, then take the ~top10 words in each topic and use that set of words to represent all of the vocabulary?

Answer 1

This is not really an answer but rather a comment (don't have the rep yet) - in general, LDA is perfect for dimensionality reduction under the bag-of-words assumption. But you wouldn't really do that the way you suggested (or at least I haven't seen it done like this). Using LDA you would estimate two things - topics (that is distributions over the vocabulary) and the document specific mixing proportions (how much of each topic is there in each of the documents?). You can now use those mixing proportions as a lower dimensional representation of your documents (instead of using full vector of term counts for example). Another solution would be to look into word embeddings such as word2vec.

Answer 2

EDIT:

So after some more research, it appears to me that what they (e.g. the SAM paper) refer to as "topic proportion" is actually alluding the normalized posterior Dirichlet parameters $\gamma^\ast(\mathbf w_d)$ for the topic distribution $\theta_d$ in document $\mathbf w_d$. That the normalized $\gamma^\ast(\mathbf w_d)$ is the expected topic proportion is due to the formula of the mean of Dirichlet distribution -- given $\boldsymbol x \sim \text{Dirichlet}(\boldsymbol x \mid \boldsymbol\alpha)$, $\mathbb E[x_k] = \alpha_k / \sum_{k'} \alpha_{k'}$. Indeed, the authors must be referring to a parameterization of the raw topic proportion since it is a random variable!

In scikit-learn (after version 0.18), you may obtain the topic proportion by:

from sklearn.decomposition import LatentDirichletAllocation

X = ...  # your data
topic_proportion = LatentDirichletAllocation().fit_transform(X)

One caveat about the snippet, though, is that I haven't confirmed the theory against scikit-learn source code.

---

You may find the answer in the original LDA paper by Blei et al. (2003). In section 7.2, the authors proposed to use the posterior Dirichlet parameters $\gamma^\ast(\mathbf w)$, i.e. the variational parameters of the topic proportion of the document $\mathbf w$, as the reduced-dimensionality features.

However, as suggested by @yassem, and also in some other topic model papers, e.g. Reisinger et al. (2010), the SAM paper, adopting directly the topic proportion is also a valid choice.

The difference between the two options is: If using the topic proportion directly as the feature, the feature vectors will live in a probability simplex $\mathbb S^{K-1}$; otherwise, the feature vectors will reside in the open cone $\mathbb R_+^K$ without other explicit constraints. $K$ is the number of topics.

Hope it helps.

Citations in this answer:

@article{blei2003latent,
    author = {Blei, David M and Ng, Andrew Y and Jordan, Michael I},
    journal = {Journal of machine Learning research},
    number = {Jan},
    pages = {993--1022},
    title = {Latent dirichlet allocation},
    volume = {3},
    year = {2003}}

@inproceedings{reisinger2010spherical,
    author = {Reisinger, Joseph and Waters, Austin and Silverthorn, Bryan and Mooney, Raymond J},
    booktitle = {Proceedings of the 27th international conference on machine learning (ICML-10)},
    organization = {Citeseer},
    pages = {903--910},
    title = {Spherical topic models},
    year = {2010}}
```