# Choosing words in a topic, which cut-off for LDA topics?

My question is what is a good cut-off threshold for LDA topics?

I used the code in this blog post Topic modeling with latent Dirichlet allocation in Python

In that code, the author shows the top 8 words in each topic, but is that the best choice?

For each topic distribution, each word has a probability and all the words probabilities add up to 1.0

I wrote this code to print out down to an epsilon threshold:

eps=0.01
for i, topic_dist in enumerate(topic_word):
wordindex=np.argsort(topic_dist)[::-1] #rev sort
w=topic_dist[wordindex] ## this is the length of all the unique words 4258
words=[np.array(vocab)[wordindex[j]] for j in range(min(n_top_words,len(wordindex))) if w[j]>eps ]
weights=['{:.3f}'.format(w[j]) for j in range(min(n_top_words,len(wordindex))) if w[j]>eps ]
print('Topic {}: {}; {}'.format(i, ', '.join(words),', '.join(weights)))


Looking at another library gensim LdaModel, it appears that LDA most likely does not originally have the probabilities sum up to 1.0 like that and they are normalized, see below:

def show_topic(self, topicid, topn=10):
topic = self.state.get_lambda()[topicid]
topic = topic / topic.sum() # normalize to probability dist
...


Running the sample code Latent Dirichlet Allocation (LDA) with Python and calling get_lambda, one can see the lambda values are sometimes above 1.0.

ldamodel.state.get_lambda()


gives:

array([ 1.48214337,  1.48168697,  0.50442377,  0.50399559,  0.50400832,
0.5047193 ,  0.50375875,  0.50376053,  1.50224118,  0.50376574,
0.5037527 ,  0.50377459,  0.50376621,  1.49831418,  1.49832577,
1.49831855,  1.49831883,  1.49831596,  1.51053093,  3.49684196,
1.49832204,  1.49832512,  0.50316907,  0.50321838,  0.50328253,
0.50319543,  0.50317986,  0.50318815,  0.50314213,  0.5031702 ,
1.49635267,  1.49634655])


What is the best eps to choose? Is it better to not to normalize the prob dist and use the original value in a cut-off? Is it best to use the max prob value in each topic and base a cut-off from that?

In my actual datasets, sometimes an eps of 0.01, actually creates a word-less topic!!

Update

Playing with different numbers of topics, I noticed that if I have 2 topics with the load_reuters data, I get this with an eps=0.01

Topic 0: ;
Topic 1: pope; 0.013


I believe that Topic 0 can be interpreted as everything else or there needs to be more topics.

arr=[]
for n in (range(2,50)):
model = lda.LDA(n_topics=n, n_iter=20, random_state=1)
model.fit(X)
topic_word = model.topic_word_  # model.components_ also works
arr.append(
(min([max(topic_word[i]) for i in range(model.n_topics)]),
max([max(topic_word[i]) for i in range(model.n_topics)])))
plt.plot(arr)
...


So looking at this chart, n is too low if below 5 and flattens out sometime after 20... A tolerance $\epsilon > 0.01$ is far too low for showing which words pertain to each topic. A primary purpose of LDA is to group words such that the topic words in each topic are highly probable within that topic. If such a low threshold is chosen, then many, many words will appear in each topic, again defeating the purpose of succinct text summarization. To extract the most probable words, you would be better off choosing a threshold of $\epsilon > 0.9$ or maybe $\epsilon > 0.8$.