Large scale text classification I am looking to do classification on my text data. I have 300 classes, 200 training documents per class (so 60000 documents in total) and this is likely to result in very high dimensional data (we may be looking in excess of 1million dimensions).
I would like to perform the following steps in the pipeline (just to give you a sense of what my requirements are):


*

*Converting each document to feature vector (tf-idf or vector space model)

*Feature selection (Mutual Information based preferably, or any other standard ones) 

*Training the classifier (SVM, Naive Bayes, Logistic Regression or Random Forest)

*Predicting unseen data based on the classifier model trained. 


So the question is what tools/framework do I use for handling such high dimensional data? I am aware of the usual suspects (R, WEKA...) but as far as my knowledge goes (I may be wrong) possibly none of them can handle data this large. Is there any other off the shelf tool that I could look at?
If I have to parallelize it, should I be looking at Apache Mahout? Looks like it may not quite yet provide the functionality I require.
Thanks to all in advance.

Update: I looked around this website, R mailing list and the internet in general. It appears to me that the following problems could emerge in my situation:
(1) Preprocessing of my data using R (tm package in particular) could be impractical, since tm will be prohibitively slow. 
(2) Since I will need to use an ensemble of R packages (pre-processing, sparse matrices, classifiers etc.) interoperability between the packages could become a problem, and I may incur an additional overhead in converting data from one format to another. For example, if I do my pre-processing using tm (or an external tool like WEKA)  I will need to figure out a way to convert this data into a form that the HPC libraries in R can read. And again it is not clear to me if the classifier packages would directly take in the data as provided by the HPC libraries. 
Am I on the right track? And more importantly, am I making sense  ?
 A: This should be possible to make it work as long as the data is represented as a sparse data structure such as scipy.sparse.csr_matrix instance in Python. I wrote a tutorial for working on text data. It is further possible to reduce the memory usage further by leveraging the hashing trick: adapt it to use the HashingVectorizer instead of the CountingVectorizer or the TfidfVectorizer. This is explained in the documentation section text features extraction.
Random Forests are in general much more expensive than linear models (such as linear support vector machines and logistic regression) and multinomial or Bernoulli naive Bayes and for most text classification problems that do not bring significantly better predictive accuracy than simpler models.
If scikit-learn ends up not being able to scale to your problem, Vowpal Wabbit will do (and probably faster than sklearn) albeit it does not implement all the models your are talking about.
Edited in April 2015 to reflect the current state of the scikit-learn library and to fix broken links.
A: Gensim for Python is magic. And since it's in Python, you can use it in conjunction with @ogrisel's suggestion.
A: Not to toot my own horn, but I made a pretty popular video series on text analytics with Rapidminer. You can see it here:
http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-loading.html
You can likely avoid doing feature selection, just use a classifier that doesn't create a million * million matrix in memory :)
Logistic regression will choke on that many dimensions. Naive Bayes assumes independent dimensions, so you will be fine. SVM doesn't depend on the number of dimensions (but on the number of support vectors) so it will be fine as well. 
300 is a lot of classes though. I would start with only a few and work your way up. 
A: First, based on your comments, I would treat this as 300 binary (yes/no) classification problems.  There are many easy-to-use open source binary classifier learners, and this lets you trade time for memory.   
SVMs and logistic regression are probably the most popular approaches for text classification.  Both can easily handle 1000000 dimensions, since modern implementations use sparse data structures, and include regularization settings that avoid overfitting.  
Several open source machine learning suites, including WEKA and KNIME, include both SVMs and logistic regression.   Standalone implementations of SVMs include libSVM and SVMlight.  For logistic regression, I'll plug BXRtrain and BXRclassify, which I developed with Madigan, Genkin, and others.  BXRclassify can build an in-memory index of thousands of logistic regression models and apply them simultaneously.  
As for converting text to attribute vector form, I somehow always end up writing a little Perl to do that from scratch. :-) But I think the machine learning suites I mentioned include tokenization and vectorization code.  Another route would be to go with more of a natural language toolkit like LingPipe, though that may be overkill for you. 
A: Since Sklearn 0.13 there is indeed an implementation of the HashingVectorizer.
EDIT: Here is a full-fledged example of such an application from sklearn docs
Basically, this example demonstrates that you can classify text on data that cannot fit in the computer's main memory (but rather on disk / network / ...).
