I have a bunch of documents and would like to to assign one (or more) topics to each document. The topics can be quite wide ranging from political topics to sports, etc. I think the best way is to deal with this problem as a classification problem however my main issue is a good and accurate training data set. The data set should be quite generic and covers diverse range of topics. I'd like to know whether there is any training data set out there for this purpose? There might be some data sets mostly for research purposes but I need an accurate one (even if it deosn't come for free).
$\begingroup$
$\endgroup$
1
-
$\begingroup$ try wikipedia: dumps.wikimedia.org/enwiki/latest $\endgroup$– Alexey GrigorevCommented Jun 25, 2015 at 8:20
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
4
You suggest that the best way is to treat the problem as a classification problem. However, in many cases topic discovery or topic modelling is treated as an unsupervised learning problem, meaning that you do not need to have the correct label for each document beforehand. A well known probabilistic model for this purpose is Latent Dirichlet Allocation. In LDA, each document is represented as a bag of words. Words in each document are assumed to be generated from a number of topics. Topics (there can be more than one) in each documents are automatically discovered along with relevant weights (probabilities). Useful resources can be found on David Blei's page.
-
$\begingroup$ I am familiar with LDA. It is useful when the documents are talking about the same topic and we would like to know what that topic(s) is. However consider a case in which you got a million document and each document is talking about a unique topic. There is no way to use LDA in this scenario. In this scenario we should first do a clustering to find similar documents and then do a LDA for each cluster which may not be possible when data set is big! $\endgroup$– HHHCommented Jun 25, 2015 at 15:02
-
$\begingroup$ You should update your question to reflect this. I did not realize you are considering a big data scenario. $\endgroup$– wijCommented Jun 25, 2015 at 18:08
-
$\begingroup$ @H.Z. You should use then online versions of the LDA algorithm like the one implemented in Vowpal Wabbit. It will happily crunch terabyte scale dataset even on a modest laptop. $\endgroup$ Commented Apr 26, 2016 at 4:11
-
$\begingroup$ @H.Z. I'm a bit confused... do you mean that each document has a unique topic (in your example, there are a million distinct topics) or that there a large number of topics (maybe 100 or 1,000) and that each document has some mixture of the topics? In the latter case, LDA will work (albeit, slowly unless you use the online version as xeon suggested). $\endgroup$ Commented Aug 3, 2016 at 14:45
$\begingroup$
$\endgroup$
Following your latest comment with regards to "each document is talking about a unique topic", I would suggest you to try the mixture-of-unigrams topic model (i.e. one topic per document). The model details with Gibbs sampling-based inference are presented in:
Jianhua Yin and Jianyong Wang. 2014. A Dirichlet Multinomial Mixture Model-based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 233–242.
http://dbgroup.cs.tsinghua.edu.cn/wangjy/papers/KDD14-GSDMM.pdf
You can also find my Java implementation for this model at http://jldadmm.sourceforge.net/
$\begingroup$
$\endgroup$
Mallet comes with a small sample corpus (some English and German texts from Wikipedia) to get the gist of it.
For learning a new tool I suggest that you first use a dataset that you already understand well.