Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite

I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single synonym. For instance, 'giant' and 'enormous' and 'huge' would all map to big. I also want to account for polysemy, where words can have more than one meaning. If I were to convert all words to their synonyms while accounting for all possible double,triple,etc. meanings, do you think that the number of distinct words in a document would increase or decrease? (Note: assume that all of the words have already been stemmed as well).

In addition, anyone have any good API suggestions for this possibly? (preferably Java)

In order to address some issues from the comments below: My database of documents consists of mainly forums from the web and links that are fetched through the forums (A web crawler was used to crawl about 10 million sites so far with many more to come). And maybe this might be a better question to ask: For anyone that has tried different methods of dimensional reduction, did you find mapping words to synonyms useful, and if you did, then how did you account for polysemy?

share|improve this question
This sounds like a question about the nature of English text. What is the statistical content of your query? – whuber♦ Aug 8 '12 at 18:05
Well this is specifically for data mining. I was hoping some data miners on here might have had similar questions before but have answers now. Many of them have to look at mapping words to synonyms to decrease the number of distinct words so the computer can process dense matrices. – Sidd Aug 8 '12 at 18:08
I don't think it's reasonable to expect a useful answer without clearly describing what kinds of documents are under consideration. At that point the question is likely to become highly specialized and probably would not be suitable for any SE site, I'm afraid (except possibly the English site: have you considered that?). If you have an actual statistical/data mining/machine learning question to ask, then we warmly welcome it and invite you to make suitable edits here. – whuber♦ Aug 8 '12 at 18:14
I asked this on the English site and they told me to try this site and Linguistics. I didn't know which one would be better so I posted on both. I apologize if this question wasn't asked properly, so I'll either try to edit it to fit your criteria or delete it. – Sidd Aug 8 '12 at 18:16
Because this is cross-posted and is collecting answers on the linguistics site, let's keep it there. – whuber♦ Aug 8 '12 at 20:03

closed as off topic by whuber♦ Aug 8 '12 at 20:03

Questions on Cross Validated are expected to relate to statistics within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.