Automating statistical correlation between “texts” and “data”

Question

I am collecting textual data surrounding press releases, blog posts, reviews, etc of certain companies' products and performance.

Specifically, I am looking to see if there are correlations between certain types and/or sources of such "textual" content with market valuations of the companies' stock symbols.

Such apparent correlations can be found by the human mind fairly quickly - but that is not scalable. How can I go about automating such analysis of disparate sources?

Answer 1

My students do this as their class project. A few teams hit the 70%s for accuracy, with pretty small samples, which ain't bad.

Let's say you have some data like this:

Return Symbol News Text
-4% DELL   Centegra and Dell Services recognized with Outsourcing Center's...
7%   MSFT   Rising Service Revenues Benefit VMWare
1%   CSCO   Cisco Systems (CSCO) Receives 5 Star Strong Buy Rating From S&P
4%   GOOG   Summary Box: Google eyes more government deals
7%   AAPL   Sohu says 2nd-quarter net income rises 10 percent on higher...

You want to predict the return based on the text.

This is called Text Mining.

What you do ultimately is create an enormous matrix like this:

Return Centegra Rising Services Recognized...
-4%    0.23     0      0.11     0.34
7%     0        0.1    0.23     0
...

That has one column for every word, and one row for each return, and a weighted score for each word. The score is generated via TFIDF or SVD method. Then you run a regression and see if you can predict which words predict the return. You'll probably need to use PCA first. Or use a machine learning method like SVM or Naive Bayes.

Book: Fundamentals of Predictive Text Mining, Weiss

Software: RapidMiner with Text Plugin or R

You should also do a search on Google Scholar and read up on the ins and outs.

You can see my series of text mining videos here

Answer 2

As per above, you need a set of articles and responses, and then you train eg. a Neural Net to them. RapidMiner will let you do this but there are many other tools out there that will let you do regressions of this size. Ideally your response variable will be consistent (ie % change after 1 hour exactly, or % change after 1 day exactly etc).

You may also want to apply some sort of filtering or classification to your training variables ie the words in the article. This could be as simple as filtering some words (eg prepositions, pronouns) or more complex like using syntax to choose which words should go into the regression. Note that any filtering you do risks biasing the result.

Some folks at University of Arizona already made a system that does this - their paper is on acm here and you may find it interesting. http://www.computer.org/portal/web/csdl/doi/10.1109/MC.2010.2 (you'll need a subscription to access if you're not eg at university). The references may also help point you in the right direction.