Text Mining: how to cluster texts (e.g. news articles) with artificial intelligence?

Question

I have built some neural networks (MLP (fully-connected), Elman (recurrent)) for different tasks, like playing Pong, classifying handwritten digits and stuff...

Additionally I tried to build some first convolutional neural networks, e.g. for classifying multi-digit handwritten notes, but I am completely new to analyze and cluster texts, e.g. in image recognition/clustering tasks one can rely on standardized input, like 25x25 sized images, RGB or greyscale and so on...there are plenty of pre-assumption features.

For text mining, for instance news articles, you have an ever changing size of input (different words, different sentences, different text length, ...).

How can one implement a modern text mining tool utilizing artificial intelligence, preferably neural networks / SOMs?

Unfortunately I were unable to find simple tutorials to start-off. Complex scientific papers are hard to read and not the best option for learning a topic (as to my opinion). I already read quite some papers about MLPs, dropout techniques, convolutional neural networks and so on, but I were unable to find a basic one about text mining - all I found was far too high level for my very limited text mining skills.

Answer 1

Latent Dirichlet Allocation (LDA) is great, but if you want something better that uses neural networks I would strongly suggest doc2vec (https://radimrehurek.com/gensim/models/doc2vec.html).

What it does? It works similarly to Google's word2vec but instead of a single word feature vector you get a feature vector for a paragraph. The method is based on a skip-gram model and neural networks and is considered one of the best methods to extract a feature vector for documents.

Now given that you have this vector you can run k-means clustering (or any other preferable algorithm) and cluster the results.

Finally, to extract the feature vectors you can do it as easy as that:

from gensim.models import Doc2Vec
from gensim.models.doc2vec import LabeledSentence

class LabeledLineSentence(object):
    def __init__(self, filename):
        self.filename = filename
    def __iter__(self):
        for uid, line in enumerate(open(self.filename)):
            yield LabeledSentence(words=line.split(), labels=['TXT_%s' % uid])


sentences = LabeledLineSentence('your_text.txt')

model = Doc2Vec(alpha=0.025, min_alpha=0.025, size=50, window=5, min_count=5,
                dm=1, workers=8, sample=1e-5)

model.build_vocab(sentences)

for epoch in range(500):
    try:
        print 'epoch %d' % (epoch)
        model.train(sentences)
        model.alpha *= 0.99
        model.min_alpha = model.alpha
    except (KeyboardInterrupt, SystemExit):
        break

Answer 2

Apart from LDA you can use Latent Semantic Analysis with K-Means. It's not neural networks, but rather "classical" clustering, but it works quite well.

Example in sklearn (taken from here):

dataset = fetch_20newsgroups(subset='all', shuffle=True, random_state=42)
labels = dataset.target
true_k = np.unique(labels).shape[0]

vectorizer = TfidfTransformer()
X = vectorizer.fit_transform(dataset.data)

svd = TruncatedSVD(true_k)
lsa = make_pipeline(svd, Normalizer(copy=False))

X = lsa.fit_transform(X)

km = KMeans(n_clusters=true_k, init='k-means++', max_iter=100)
km.fit(X)

Now cluster assignment labels are available in km.labels_

For example, these are the topics extracted from 20 newsgroups with LSA:

Cluster 0:  space  shuttle  alaska  edu  nasa  moon  launch  orbit  henry  sci
Cluster 1:  edu  game  team  games  year  ca  university  players  hockey  baseball
Cluster 2:  sale  00  edu  10  offer  new  distribution  subject  lines  shipping
Cluster 3:  israel  israeli  jews  arab  jewish  arabs  edu  jake  peace  israelis
Cluster 4:  cmu  andrew  org  com  stratus  edu  mellon  carnegie  pittsburgh  pa
Cluster 5:  god  jesus  christian  bible  church  christ  christians  people  edu  believe
Cluster 6:  drive  scsi  card  edu  mac  disk  ide  bus  pc  apple
Cluster 7:  com  ca  hp  subject  edu  lines  organization  writes  article  like
Cluster 8:  car  cars  com  edu  engine  ford  new  dealer  just  oil
Cluster 9:  sun  monitor  com  video  edu  vga  east  card  monitors  microsystems
Cluster 10:  nasa  gov  jpl  larc  gsfc  jsc  center  fnal  article  writes
Cluster 11:  windows  dos  file  edu  ms  files  program  os  com  use
Cluster 12:  netcom  com  edu  cramer  fbi  sandvik  408  writes  article  people
Cluster 13:  armenian  turkish  armenians  armenia  serdar  argic  turks  turkey  genocide  soviet
Cluster 14:  uiuc  cso  edu  illinois  urbana  uxa  university  writes  news  cobb
Cluster 15:  edu  cs  university  posting  host  nntp  state  subject  organization  lines
Cluster 16:  uk  ac  window  mit  server  lines  subject  university  com  edu
Cluster 17:  caltech  edu  keith  gatech  technology  institute  prism  morality  sgi  livesey
Cluster 18:  key  clipper  chip  encryption  com  keys  escrow  government  algorithm  des
Cluster 19:  people  edu  gun  com  government  don  like  think  just  access

You also can apply Non-Negative Matrix Factorization, which can be interpreted as clustering. All you need to do is to take largest component of each document in the transformed space - and use it as cluster assignment.

In sklearn:

nmf = NMF(n_components=k, random_state=1).fit_transform(X)
labels = nmf.argmax(axis=1)

Answer 3

LSA+KMeans works well but you have to input the amount of clusters you are expecting. Moreover the silhouette coefficient of the clusters found is usually low.

Another method with which I get better results is to use DBSCAN example here. It search for centers of high density and expands to make clusters. In this method it automatically finds the optimal amount of clusters.

I've also found it very important to use a stemmer, such as Snowball for ex, which reduces the errors due to typos. A good stop words list is also very important if you want to be sure to get rid of some clusters which would have no meaning because of the high occurrence of common words with no significant meaning. When you build your count matrix, normalisation is also important, it allows to add weigh to a word with a low occurrence on the dataset, but with high occurrence in particular samples. These words are meaningful and you don't want to miss them. It also lowers weights of words with high occurrences in all particular samples (close to stop word but for words which can have a little meaning). One last thing I noticed was important is not to print the top 10 words of your clusters, but a more extended selection. Usually the quality and relevance of the keywords toward the label you would give to the cluster tend to reduce dramatically after these top 10-20 words. So an extended view of top keywords will help you to analyze if your cluster is really relevant or very polluted by noise.

Answer 4

My favorite method is LDA; you can look here for a tutorial using python packages.

You can also look at much simpler methods like this.