I am confused as to how to interpret the LDA's perplexity fluctuations with different numbers of topics, in the endeavour of determining the best number of topics. Additionally, I would like to know how to implement AIC/BIC with gensim LDA models.
I am importing the 20 newsgroups dataset from sklearn:
from sklearn.datasets import fetch_20newsgroups;
Metadata were removed as per sklearn recommendation, and the data were split to test and train using sklearn also (
I trained 35 LDA models with different values for k, the number of topics, ranging from 1 to 100, using the train subset of the data. Afterwards, I estimated the per-word perplexity of the models using gensim's multicore LDA
log_perplexity function, using the test held-out corpus::
DLM_testCorpusBoW = [DLM_fullDict.doc2bow(tstD) for tstD in testData];
PerWordPP = modelLDA.log_perplexity(DLM_testCorpusBoW);
Eventually, keeping in mind that true k is 20 for the used dataset, perplexity figures were startling negative ones:
Additionally, I implemented Topic Coherence Models, and the results weren't very informative, with a lot of fluctuation:
Having negative perplexity apparently is due to infinitesimal probabilities being converted to the log scale automatically by Gensim, but even though a lower perplexity is desired, the lower bound value denotes deterioration (according to this), so the lower bound value of perplexity is deteriorating with a larger number of topics in my figures, yet we expect the perplexity to improve with a bigger k (here).
So, do the figures seem logical? And I would like to implement AIC and BIC as well, but for that I need the SSE of the models, how can I get it using gensim? or are these measures implemented somehow?