Determining trends in text I have a bunch of documents (66 quarterly reports on grievances and complaints on health care services provided and growing) and a list of words that I'd like to follow over time.  What is the easiest way to do this?  I have played with R's text mining library and have got rather frustrated.  I have also tried to use RapidMiner and it chokes on three documents (ran out of memory).  I would greatly appreciate any suggestions, ideas etc...
 A: There are a number of statistical NLP projects out there, with NLTK one of the more active Open Source ones. However tracking word frequency over time and over a few hundred documents is probably a simple enough problem to code yourself.


*

*You will want to start with converting your documents to a format easy to process, like plain text ala your comment. Convert to lower case, drop punctuation, and then split each document into words. Start with a regular expression like /\b/, and then filter out numbers and obvious errors.

*Next you will probably want to drop stop words. Here is a decent stop word list for English language sources.

*Now count each occurrence of a word in each document. You will probably want to build a hashtable index, with (non-stop) word as the key and the value as an integer count.

*If you would like to get more sophisticated, you could pull out collocations by adding each preceding or succeeding $n-1$ words to your index. Or even run each word through a stemmer like this Ruby gem.

*Lastly sort your words by index count.


Here are your steps for the plain text document: The quick brown fox jumps over the quick, brown lazy dog.


*

*Convert to lower case, and drop the puncutation: the quick brown fox jumps over the quick brown lazy dog

*Split into words with /\b/, drop the words that are all whitespace: the; quick; brown; fox; jumps; over; the; quick; brown; lazy; dog

*Now drop the stop words: quick; brown; fox; jumps; over; quick; brown; lazy; dog

*Build your count index: quick=2; brown=2; fox=1; jumps=1; over=1; lazy=1; dog=1

*Add 2-gram collocations: quick-brown=2; quick=2; brown=2; brown-fox=1; fox=1; fox-jumps=1; jumps-over=1...

*Stem the words, so jumps and jumped become just jump.

A: Text Mining Process Overview
Here is a quick summary of generic approach/process for identifying trends in text. There are great open source tools available (R, python, etc) to carry out the process mentioned here.

Brief Description Of Data Processing Steps
Explore Corpus - Understand the types of variables, their functions, permissible values, and so on. Some formats including html and xml contain tags and other data structures that provide more metadata.
Convert text to lowercase - This is to avoid distinguish between words simply on case.
Remove Number(if required) - Numbers may or may not be relevant to our analyses.
Remove  Punctuations - Punctuation can provide grammatical context which supports understanding. Often for initial analyses we ignore the punctuation.  Later we will use punctuation to support the extraction of meaning.
Remove  English stop words - Stop words are common words found in a language. Words like for, very, and, of, are, etc, are common stop words.
Remove  Own stop words(if required) - Along with English stop words, we could instead or in addition remove our own stop words. The choice of own stop word might depend on the domain of discourse, and might not become apparent until we've done some analysis.
Strip white space - Eliminate extra white spaces.
Stemming - Stemming uses an algorithm that removes common word endings for English words, such as “es”, “ed” and “'s”.
Sparse terms - We are often not interested in infrequent terms in our documents. Such “sparse" terms should be removed from the document term matrix.
Document term matrix - A document term matrix is simply a matrix with documents as the rows and terms as the columns and  a count of the frequency of words as the cells of the matrix.
References & R Example Codes

*

*https://github.com/benmarwick/AAA2011-Tweets

*http://www.rdatamining.com/docs

*http://onepager.togaware.com/TextMiningO.pdf

*http://www.rtexttools.com/about-the-project.html
A: It sounds to me that you might be talking about performing sentiment analysis (if not, the previous two answers are very good, but skip to the last paragraph below). If that is the case, you I have some suggestions for you. I would recommend you to start from reading the draft of the introductory book "Sentiment analysis and opinion mining" by Bing Liu. The draft in a PDF document format is available for free here. More details about the new upcoming book of this author, as well as comprehensive information on the topic of aspect-based sentiment analysis, with references and links to data sets, are available at this page.
Another interesting resource is a survey book "Opinion mining and sentiment analysis by Bo Pang and Lillian Lee. The book is available in print and as a downloadable PDF e-book in a published version or an author-formatted version, which are almost identical in terms of contents.
Finally, speaking about software tools for NLP, in general, in addition to NLTK and other already recommended tools, I highly recommend you to evaluate the Stanford NLP Group's open source software (http://www-nlp.stanford.edu/software). Should you need a more scalable solution in the future, you might want to take a look at another interesting set of open source libraries - parallel framework for machine learning GraphLab (http://select.cs.cmu.edu/code/graphlab). It is especially suitable for very large volume of data, as it implements MapReduce model and, thus, supports multicore and multiprocessor parallel processing.
