What is the most efficient way of training data using least memory? This is my training data: 200,000 Examples x 10,000 Features.
So my training data matrix is - 200,000 x 10,000. 
I managed to save this in a flat file without having memory issues by saving every dataset one by one (one example after another) as I generate the features for each example.
But, now when I use Milk, SVMlight, or any other Machine Learning Algorithm, everything tries to load the whole training data into memory instead of training it one by one. However I just have 8 GB RAM, so I cannot proceed this way. 
Do you know of anyway I could train the algorithm one dataset by one dataset? I.e., so that at any instant I just have one dataset loaded into memory, while training. 
 A: I believe the term for this type of learning is out-of-core learning.  One suggestion is vowpal wabbit, which has a convenient R library, as well as libraries for many other languages.
A: I heartily second Zach's suggestion.  vowpal wabbit is an excellent option, and you'd be suprised by its speed.  A 200k by 10k data-set is not considered large by vowpal wabbit's norms.
vowpal_wabbit (available in source form via https://github.com/JohnLangford/vowpal_wabbit, an older version is available as a standard package in Ubuntu universe) is a fast online linear + bilinear learner, with very flexible input.
You may mix binary and numeric-valued features.  There's no need to number the features as variable names will work "as is".  It has a ton of options, algorithms, reductions, loss-functions, and all-in-all great flexibility.  You may join the mailing list (find it via github) and ask any question. The community is very knowledgable and supportive.
A: I answered similar question here. Point is most machine learning/data mining algorithms are batch learners that is they load all data to memory. Therefore you need different tools for very large data sets as you have. See that questions's tools also.
Online Learning is a way to reduce memory footprint of algorithms.
