Tag Info

Recently, a huge body of literature discussing how to extract information from written text has grown. Hence I will just describe four milestones/popular models and their advantages/disadvantages and thus highlight (some of) the main differences (or at least what I think are the main/most important differences). You mention the "easiest" approach, which ...

I think it is not really a question of better and worse but what data you have available and interpretability. If the data you have is at least partially labeled, whether with something like traditional topic classes, or something like hashtags, then labeled LDA may be interesting to pursue, otherwise not. Doing labeled LDA is certain to mean that the ...

You are trying to solve two problems here. Problem 1: Categorize questions strings in the proper category. Problem 2: Create proper categories. The first problem could be done by so-called supervised algorithms, many classifiers can give very good accuracy and performance. However, problem 2, creating categories out of thin air (tons of data), is much ...

While I'm not super familiar with his work, I know Jacob Eisenstein has done work in text analysis and graphical models in twitter data. In particular, this paper describes an application of topic modeling in twitter data and microblogs. Edit: actually after reading the paper a bit more, they state: However, the average message on Twitter is only ...

This is a fairly standard problem in NLP, and the magic Google words you're looking for are "topic modeling". Although your strings are quite short, you may have some success with Latent Dirichlet Allocation, or a similar method. There's a nice blog post by Edwin Chen here, which lays out the general idea behind the algorithm. The details of implementation ...

I tried to follow authors' instructions with a very simple test case. Here is the R code that I used. set.seed(123) # Create the data according to authors' notations. # For the first 100 observations y = 1, and for the last 100 y = 0. # Only the first 25 obersvations are labelled, according to their # "chosen completely at random" criterion. c <- 25/100 ...

Weka offers this functionality in Java. Start Weka and open the Explorer. Then load your dataset and apply the StringToWordVector filter. This filter can create a doc term matrix (either binary or by frequency), do IDF, stopword removal, stemming, normalization, punctuation removal and more.

Python NLTK has a dataset called hmm_treebank_pos_tagger that you can download here. Stanford has a POS tagger described here. You can download it along with the training data