1
$\begingroup$

We use topic modelling usually on a collection of documents - which makes the input. But what if I only have a single document where I want to see the underlying topics in it? I have heard that you can break them by paragraphs in cases like that, but what is the need for that? Does that mean I can't use latent dirichlet allocation (LDA) or it is not supposed to use with a single document as the input?

$\endgroup$
2
$\begingroup$

You can use a sentence splitter and split your document into sentences. I have never used the approach myself, but the tool is available with the open.nlp package in R, Python and Rapidminer.

What you could also do is to train a topicmodel on corpus with clearly defined topics. Next you use the same model on your one document and you see how the topic structure turn out.

$\endgroup$
  • $\begingroup$ thanks! but second option is not available to me so it's out. You can split them by sentence but then again the document length will be small which makes LDA to find document level word co-occurrences due to the sparsity. What's wrong with using a single document as the input to LDA? $\endgroup$ – samsamara Dec 18 '14 at 0:00
  • 1
    $\begingroup$ If you only use a single document you only get one observation. I cannot see how that should work... The topic model are derived based on word co-occurrences across n documents. I am having a very hard time imagine that should make sense in any way... You can do a wordcloud on the one document and filter out stopwords and short words etc. Maybe that is better... $\endgroup$ – Kasper Christensen Dec 18 '14 at 9:27
0
$\begingroup$

In my own experience, using the gensim lda model, it worked out just fine with only one document. Gensim's lda requires a list of "corpus". My list only contains one element. The results, tested on a number of documents, looked pretty good.

It is not the most sophisticated strategy, as it is solely count-based, however, it does the job and you cannot really get more out of it if there's no further input for the model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.