Besides LDA (Latent Dirichlet allocation), are there other ways or methods to detect a topic or category from a sentence?

For example, all the categories or tags from news websites can be used to train a classifier to predict the topic for a new sentence or article or a paragraph.

Are there any known public datasets with keywords to topics relationship which can be used a training set in a classic classification problem?

A simple training data example of a news website:

Article 1: category x 
Article 2: category x 
Article 3: category x
Article 4: category y 
Article 5: category y 
Article 6: category y

It's a very simple example but it's enough to paint the picture. Now use this data to predict the category of a new article.

A bit of explanation: By topic I mean if a text is talking about politics, entertainment, business, finance, lifestyle etc and ideally, a classification into sub-categories of such types. A similar categorization is used in the news website where they place each article in a specific category.


2 Answers 2


It sounds like you are talking about a text classification problem. There are lots of data sets for this.

  1. 20 newsgroups data http://qwone.com/~jason/20Newsgroups/

  2. Yahoo answers data https://github.com/juanshishido/text-classification

  3. DBPedia ontology dataset (can't find a link right now)

There is also something called supervised LDA that might interest you.

  • $\begingroup$ 1 and 2 are relevant but too small for my use case. $\endgroup$
    – utengr
    Sep 26, 2017 at 8:35
  • $\begingroup$ You could do hashtag prediction on Twitter $\endgroup$
    – Aaron
    Sep 26, 2017 at 15:55

I really liked Gensim package for Python. There they use a collection of articles from wikipedia as training, you can find it here (8GB).

As transformations they use:

  1. Term Frequency * Inverse Document Frequency, Tf-Idf
  2. Latent Semantic Indexing, LSI (or sometimes LSA)
  3. Random Projections, RP

and then LDA and its transformation HDP. You can find a good tutorial here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.