# Software or libraries to create doc-term matrix

does anyone know some Java libraries to create the document-term matrix for a large number (50,000) of documents ? I wish this library encompasses preprocessing functionalities, like stop-word and punctuation removal, stemming, etc. What's more, I wish to use TF*IDF weighting scheme, and normalization functionality. Preferably, I prefer Java libraries for the convenience of development.

Thanks very much for any recommendation.

-

Weka offers this functionality in Java. Start Weka and open the Explorer. Then load your dataset and apply the StringToWordVector filter. This filter can create a doc term matrix (either binary or by frequency), do IDF, stopword removal, stemming, normalization, punctuation removal and more.

-
After my experiment, I think Weka will be suitable for not very huge sizes (several hundred MB txt data). Otherwise, weka will throw a "java heap size" problem, because it stores all texts for preprocessing and their statistics in RAM. For more code, perhaps, interested readers may refer to here Last but not least, Weka is the best for documentation among all proposed solutions! Thanks! –  Ensom Hodder Jun 25 '12 at 15:02
Good to hear that it suits your needs. About the heap size problem; You can set the heap size of Java to a higher value when you run Weka (see weka.wikispaces.com/OutOfMemoryException). –  Sicco Jun 25 '12 at 17:27
:-) of course i know how to increase the Java Heap size, but Weka's strategy to hold all files and their info in RAM is not feasible when I use 100K files to preprocess. What's worse, my file are more than 10K words per file. So I am afraid i have to figure another "intelligent" and tricky method to handle this :-). Anyway, thanks a lot for your help ! –  Ensom Hodder Jun 25 '12 at 18:09

JFreq might do what you want. There's a command line version too.

-
Hum, it seems that this software meeting my requirements perfectly. I will look into it, and hopefully, I can plug it as a module in my application. –  Ensom Hodder Jun 22 '12 at 14:52
I am afraid this tool is not suitable for redevelopment (integrate in your own software), and also, lack of documentation for development work :-( –  Ensom Hodder Jun 25 '12 at 8:20
It's true that it's more an application rather than a library. You probably want something like Mallet then. –  conjugateprior Jun 25 '12 at 10:08