3
$\begingroup$

does anyone know some Java libraries to create the document-term matrix for a large number (50,000) of documents ? I wish this library encompasses preprocessing functionalities, like stop-word and punctuation removal, stemming, etc. What's more, I wish to use TF*IDF weighting scheme, and normalization functionality. Preferably, I prefer Java libraries for the convenience of development.

Thanks very much for any recommendation.

$\endgroup$
3
$\begingroup$

Weka offers this functionality in Java. Start Weka and open the Explorer. Then load your dataset and apply the StringToWordVector filter. This filter can create a doc term matrix (either binary or by frequency), do IDF, stopword removal, stemming, normalization, punctuation removal and more.

$\endgroup$
3
  • $\begingroup$ After my experiment, I think Weka will be suitable for not very huge sizes (several hundred MB txt data). Otherwise, weka will throw a "java heap size" problem, because it stores all texts for preprocessing and their statistics in RAM. For more code, perhaps, interested readers may refer to here Last but not least, Weka is the best for documentation among all proposed solutions! Thanks! $\endgroup$ Jun 25 '12 at 15:02
  • $\begingroup$ Good to hear that it suits your needs. About the heap size problem; You can set the heap size of Java to a higher value when you run Weka (see weka.wikispaces.com/OutOfMemoryException). $\endgroup$
    – Sicco
    Jun 25 '12 at 17:27
  • 1
    $\begingroup$ :-) of course i know how to increase the Java Heap size, but Weka's strategy to hold all files and their info in RAM is not feasible when I use 100K files to preprocess. What's worse, my file are more than 10K words per file. So I am afraid i have to figure another "intelligent" and tricky method to handle this :-). Anyway, thanks a lot for your help ! $\endgroup$ Jun 25 '12 at 18:09
3
$\begingroup$

JFreq might do what you want. There's a command line version too.

$\endgroup$
3
  • $\begingroup$ Hum, it seems that this software meeting my requirements perfectly. I will look into it, and hopefully, I can plug it as a module in my application. $\endgroup$ Jun 22 '12 at 14:52
  • $\begingroup$ I am afraid this tool is not suitable for redevelopment (integrate in your own software), and also, lack of documentation for development work :-( $\endgroup$ Jun 25 '12 at 8:20
  • $\begingroup$ It's true that it's more an application rather than a library. You probably want something like Mallet then. $\endgroup$ Jun 25 '12 at 10:08
2
$\begingroup$

I would vote for http://lucene.apache.org/ and http://mahout.apache.org/.

$\endgroup$
1
  • $\begingroup$ Thanks for your reply. It seems lucene is a toolkit for search engine, while mahout includes some implementations for large-scale machine learning algorithms. I found both of them lack sufficient documentations, and also examples. :-( $\endgroup$ Jun 22 '12 at 14:24
0
$\begingroup$

Cleartk http://code.google.com/p/cleartk/ is a package built on top of Apache UIMA. Configuration is a bit tricky, but it works.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.