Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite
1

does anyone know some Java libraries to create the document-term matrix for a large number (50,000) of documents ? I wish this library encompasses preprocessing functionalities, like stop-word and punctuation removal, stemming, etc. What's more, I wish to use TF*IDF weighting scheme, and normalization functionality. Preferably, I prefer Java libraries for the convenience of development.

Thanks very much for any recommendation.

share|improve this question

4 Answers

up vote 2 down vote accepted

Weka offers this functionality in Java. Start Weka and open the Explorer. Then load your dataset and apply the StringToWordVector filter. This filter can create a doc term matrix (either binary or by frequency), do IDF, stopword removal, stemming, normalization, punctuation removal and more.

share|improve this answer
After my experiment, I think Weka will be suitable for not very huge sizes (several hundred MB txt data). Otherwise, weka will throw a "java heap size" problem, because it stores all texts for preprocessing and their statistics in RAM. For more code, perhaps, interested readers may refer to here Last but not least, Weka is the best for documentation among all proposed solutions! Thanks! – Ensom Hodder Jun 25 '12 at 15:02
Good to hear that it suits your needs. About the heap size problem; You can set the heap size of Java to a higher value when you run Weka (see weka.wikispaces.com/OutOfMemoryException). – Sicco Jun 25 '12 at 17:27
1  
:-) of course i know how to increase the Java Heap size, but Weka's strategy to hold all files and their info in RAM is not feasible when I use 100K files to preprocess. What's worse, my file are more than 10K words per file. So I am afraid i have to figure another "intelligent" and tricky method to handle this :-). Anyway, thanks a lot for your help ! – Ensom Hodder Jun 25 '12 at 18:09
up vote 3 down vote

JFreq might do what you want. There's a command line version too.

share|improve this answer
Hum, it seems that this software meeting my requirements perfectly. I will look into it, and hopefully, I can plug it as a module in my application. – Ensom Hodder Jun 22 '12 at 14:52
I am afraid this tool is not suitable for redevelopment (integrate in your own software), and also, lack of documentation for development work :-( – Ensom Hodder Jun 25 '12 at 8:20
It's true that it's more an application rather than a library. You probably want something like Mallet then. – conjugateprior Jun 25 '12 at 10:08
up vote 1 down vote

I would vote for http://lucene.apache.org/ and http://mahout.apache.org/.

share|improve this answer
Thanks for your reply. It seems lucene is a toolkit for search engine, while mahout includes some implementations for large-scale machine learning algorithms. I found both of them lack sufficient documentations, and also examples. :-( – Ensom Hodder Jun 22 '12 at 14:24
up vote 0 down vote

Cleartk http://code.google.com/p/cleartk/ is a package built on top of Apache UIMA. Configuration is a bit tricky, but it works.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.