How can I combine LDA with binary logistic regression by adding conditions on the topics extracted by LDA?

Question

I'd like to combine LDA (latent dirichlet allocation) and logistic regression meaning I'd like to use LDA to generate millions of documents and establish a likelihood to estimate latent variables in the data and simultaneously in the likelihood I also incorporate the classification problem such as description given in the following:

• word A and B are both in a single document but they belong to different topics (complement).
• word A is in a document but word B is in another document while both having the same topics (substitute).
• If it is possible, weight each word based on a vector of features which might be able to capture some degrees of similarities between words.

Can anybody suggest a way to formulate this problem?

I have another question if we know priori the topics and the words belong to each topic e.g. all the words in the dictionary, does it make sense to still use LDA to generate the document and then train them based on above mentioned problem?

Answer 1

LDA should group words together according to topic similarity. From what I understand it very similar to K means. I have tried to make a python code for it(it has its problems). In my code I have two matrices, a and b. Matrix a represents how much a document aligns to each topic. Matrix b represents how much a word aligns to each topic.

• word A and B are both in a single document but they belong to different topics (complement).

It should be possible to loop through all the words in a document and print the pairs which have different alignments.

• word A is in a document but word B is in another document while both having the same topics (substitute).

Again, it should be quite easy to go through the list of all words in a particular topic and then print the pairs of documents.

• If it is possible, weight each word based on a vector of features which might be able to capture some degrees of similarities between words.

My matrix b does just that.

I have another question if we know priori the topics and the words belong to each topic e.g. all the words in the dictionary, does it make sense to still use LDA to generate the document and then train them based on above mentioned problem?

I think it would be like initializing the variables in K-means clustering. It would help in faster convergence in case you then train it on a larger data set.

Further reading: http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-allocation/

Here is a nice youtube video