How to perform text mining, sentiment mining, and business category identification, and where to obtain a categorization library

Currently I am working with Text Mining which includes sentiment identification and assigning corresponding business categories using open source tool R. I found these two documents which helped me to some extent:

My approach is to tokenize the text and then lookup for sentiment and business category. To do this I require positive and negative libraries for sentiment mining and category file which contains word and category. I was able to get positive and negative words but was unable to get a categorization library.

• Where can I get a categorization library?
• Is the above approach appropriate? Is there a better way to do this?

One solution mentioned by Jeffrey Breen is to use Lu and Hiu's lexicon. He also gives a cool tutorial for sentiment mining on Twitter.

A few alternatives

1) Supervised learning: Ideally you'd want to have in your data the text and a label, where the label refers to the category you are interested. For this you'd need to manually label the data. Then you can train a statistical/machine learning algorithm in order to classify the categories you've labeled. For this a simple approach in R is using the text2vec and glmnet packages.

2) Unsupervised learning: Another alternative if you are not willing to label the data, you can fit a statistical model to find topics, this is called latent dirichlet allocation. It's also supported by text2vec, there are other packages for this such as topicmodels.

3) Transfer learning: Finally, another option is transfer learning. The idea is you get labeled data from another problem, train a classifier and use the model to make predictions in your data. The problem is finding data of the same domain...

4) Using a dictionary (hack): Below is the original answer I wrote that is using a look-up table of a "dictionary" of positive and negative words. I'd now say this option is the last resource, it's sort of a hack as it too broad. I think there are packages that implement this now.

EDIT

I've used Jeffrey's function to compute the score using the "look up" method.

I made a corruption index of polititians from Argentina. So the more people said they were corrupt, the higher the score.

For the data I had, as the tweets were in spanish, I just wrote a small dictionary more coherent with the needs of the problem. I guess it depends what you want to analyze.

Besides this approach, there are more sophisticated methods. I think there is a coursera course about NLP.

Here is another resource: https://sites.google.com/site/miningtwitter/basics/text-mining It says it's deprecated because of the changes of twitter API, but the author of twitteR package adapted the package to the new API. So you need first to install the updated package, perhaps the source version of it. Here is the author's web page. _http://geoffjentry.hexdump.org/ The later is needed in order not to get duplicated tweets when making a call bigger than 100 tweets.

Jeffrey's Function:

score.sentiment = function(sentences, pos.words, neg.words, .progress='none')
{
require(plyr)
require(stringr)

# we got a vector of sentences. plyr will handle a list
# or a vector as an "l" for us
# we want a simple array ("a") of scores back, so we use
# "l" + "a" + "ply" = "laply":
scores = laply(sentences, function(sentence, pos.words, neg.words) {

# clean up sentences with R's regex-driven global substitute, gsub():
sentence = gsub('[[:punct:]]', '', sentence)
sentence = gsub('[[:cntrl:]]', '', sentence)
sentence = gsub('\\d+', '', sentence)
# and convert to lower case:
sentence = tolower(sentence)

# split into words. str_split is in the stringr package
word.list = str_split(sentence, '\\s+')
# sometimes a list() is one level of hierarchy too much
words = unlist(word.list)

# compare our words to the dictionaries of positive & negative terms
pos.matches = match(words, pos.words)
neg.matches = match(words, neg.words)

# match() returns the position of the matched term or NA
# we just want a TRUE/FALSE:
pos.matches = !is.na(pos.matches)
neg.matches = !is.na(neg.matches)

# and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():
score = sum(pos.matches) - sum(neg.matches)

return(score)
}, pos.words, neg.words, .progress=.progress )

scores.df = data.frame(score=scores, text=sentences)
return(scores.df)
}

• Welcome to the site, @MartinBel. Is this meant as an answer to the OP's question? Would you mind fleshing out the connections between your post & the original question? I think this has a lot of potential, but it might help if it where a little more detailed. – gung Jul 6 '13 at 1:06
• @gung if you follow the link of Jeffrey Breen's blogspot posted in the first answer, it will make more sense. jeffreybreen.wordpress.com/2011/07/04/… He provides an R function to calculate the score of tweets using a Lu and Hiu's lexicon. The function is great, however the lexicon could be changed. – marbel Jul 6 '13 at 1:26
• Thanks, that helps a little. 1 of our goals is to create a permanent repository of information about statistics & machine learning. Thus, 1 thing we worry about is linkrot. We don't want your answer to become less useful should those slides disappear. Would you mind including some supporting detail so that a future reader wouldn't need to have seen that to get full value from your post? Since you're new here, you may want to read our about & help pages, which discuss things like this. – gung Jul 6 '13 at 1:36