4
$\begingroup$

I own around 40,000 text files for preprocessing (in purpose of document classification). I used R (with tm package) for prototype and now looking for a equivalent Java library for products.

However, for very fundamental tasks, i.e. text preprocessing, I found a very strange problem. That, with Weka, I apply punctuation and stop-words removal, and the same operations with R. Basically, the generated vocabulary (terms) size should be relatively the same. However, weka returns a vocabulary (attributes in arff file) with only 35,000 terms, while in R, there are more than 1 million distinct terms.

Can anyone help me understand this problem, or introduces me some more reliable Java libraries for text preprocessing?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Did you apply the StringToWordVector in Weka? If so, then you did more than just punctuation and stop-words removal. StringToWordVector outputs only the doc-term matrix of the input text files, so once the above mentioned preprocessing is done Weka will create 1 term for each unique word. 35k terms sounds logical for 40k texts.

The preprocessing in R seems to have been only the punctuation and stop-words removal. So 40k documents results in 1M words, but not unique words. Are your text files approximately 25 words on average? If this is not the case, then there is something else going on indeed.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I would say your hypothesis is not my case. For me, every document is larger than 1KB, and in my opinion, for TM package in R, when it constructs the doc-term matrix, all terms are indeed unique ! So I think Weka has do something more than expected, but I don't know what it is exactly . $\endgroup$
    – Ensom Hodder
    Jun 25, 2012 at 19:17
  • $\begingroup$ I see. Could you specify which R and Weka commands/actions you performed? $\endgroup$
    – Sicco
    Jun 25, 2012 at 22:56
  • $\begingroup$ I can hardly paste the full code in the comment here, so I explain you briefly. For R, I create VCorpus object which holds all 40K texts, then I use tm_map function with tolower,removePunctuation,removeWords filters. As to Weka, I create an Instances object which contains all texts, and a StringToWordVector filter which set usestoplist true, and also give options (L,C), which means to lowercase and output word counts rather than boolean word presence. That's all operations I take. I don't see big differences. How about you ? $\endgroup$
    – Ensom Hodder
    Jun 26, 2012 at 9:29
  • $\begingroup$ I am guessing is it possible that tm package and Weka use different tokenization algorithms, which results in different vocabulary size. What do you think of this? $\endgroup$
    – Ensom Hodder
    Jun 26, 2012 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.