This was inspired by Efficient online linear regression, which I found very interesting. Are there any texts or resources devoted to large-scale statistical computing, by which computing with datasets too large to fit in main memory, and perhaps too varied to effectively subsample. For example, is it possible to fit mixed effects models in an online fashion? Has anyone looked into the effects of replacing the standard 2nd order optimization techniques for MLE with 1st order, SGD-type techniques?
You might look into the Vowpal Wabbit project, from John Langford at Yahoo! Research . It is an online learner that does specialized gradient descent on a few loss functions. VW has some killer features:
- Installs on Ubuntu trivially, with "sudo apt-get install vowpal-wabbit".
- Uses the hashing trick for seriously huge feature spaces.
- Feature-specific adaptive weights.
- Most importantly, there is an active mailing list and community plugging away on the project.
The Bianchi & Lugosi book Prediction, Learning and Games gives a solid, theoretical foundation to online learning. A heavy read, but worth it!