# Topic detection for a sentence or an article using Machine learning

Besides LDA (Latent Dirichlet allocation), are there other ways or methods to detect a topic or category from a sentence?

For example, all the categories or tags from news websites can be used to train a classifier to predict the topic for a new sentence or article or a paragraph.

Are there any known public datasets with keywords to topics relationship which can be used a training set in a classic classification problem?

A simple training data example of a news website:

Article 1: category x
Article 2: category x
Article 3: category x
Article 4: category y
Article 5: category y
Article 6: category y


It's a very simple example but it's enough to paint the picture. Now use this data to predict the category of a new article.

A bit of explanation: By topic I mean if a text is talking about politics, entertainment, business, finance, lifestyle etc and ideally, a classification into sub-categories of such types. A similar categorization is used in the news website where they place each article in a specific category.

It sounds like you are talking about a text classification problem. There are lots of data sets for this.

1. 20 newsgroups data http://qwone.com/~jason/20Newsgroups/

3. DBPedia ontology dataset (can't find a link right now)

There is also something called supervised LDA that might interest you.

• 1 and 2 are relevant but too small for my use case. Sep 26 '17 at 8:35
• You could do hashtag prediction on Twitter Sep 26 '17 at 15:55

I really liked Gensim package for Python. There they use a collection of articles from wikipedia as training, you can find it here (8GB).

As transformations they use:

1. Term Frequency * Inverse Document Frequency, Tf-Idf
2. Latent Semantic Indexing, LSI (or sometimes LSA)
3. Random Projections, RP

and then LDA and its transformation HDP. You can find a good tutorial here.