I was wondering something for a computer science project I am doing. Documents usually have too many distinct words to process on a computer efficiently. One solution is to map all words to a single synonym. For instance, 'giant' and 'enormous' and 'huge' would all map to big. I also want to account for polysemy, where words can have more than one meaning. If I were to convert all words to their synonyms while accounting for all possible double,triple,etc. meanings, do you think that the number of distinct words in a document would increase or decrease? (Note: assume that all of the words have already been stemmed as well).
In addition, anyone have any good API suggestions for this possibly? (preferably Java)
In order to address some issues from the comments below: My database of documents consists of mainly forums from the web and links that are fetched through the forums (A web crawler was used to crawl about 10 million sites so far with many more to come). And maybe this might be a better question to ask: For anyone that has tried different methods of dimensional reduction, did you find mapping words to synonyms useful, and if you did, then how did you account for polysemy?