Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm actually working on an information retrieval project, and I want to extract words that are of significance from a sentence. This is somewhat opposite to stopwords.

In a sentence like: He was a great astrologer and an orthodontist.

... I want to extract out words that might be of more importance. To do this, I would definitely need to filter out the stop words, but words like 'orthodontist' and 'astrologer' are far more important since they do not occur frequently. I want to be able to extract such words for my training data.

Also, my need would be satisfied if there is some API that gives you the frequency of all words that are seen in a huge enough corpus (maybe a few million sentences?) that can probably represent the distribution of words in communication in English language.

For ex: I'd input 'astrologer' and I'd get a frequency score of 0.3 (this is random, and normalized) as opposed to getting a frequency score of 0.9 for 'the'

share|improve this question

closed as off topic by Peter Flom, Andy W, Macro, Peter Ellis, whuber Nov 4 '12 at 20:41

Questions on Cross Validated are expected to relate to statistics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Off topic I think, unless you want to ask a question about how create your normalised score from the raw frequencies. But is of some help? – Peter Ellis Oct 31 '12 at 19:52
up vote 1 down vote accepted

You can go for BNC dataset (British National Corpus) and it has documents of different genres. RASP4UIMA can be used to parse the document for syntactic information. PS: Downside with BNC is that it won't have newly coined terms.

share|improve this answer

In this link: there are 3 stoplists (portuguese, english and spanish).

share|improve this answer

I just discovered this resource yesterday, which may have something to help you out. Aside from that, it seems like the Natural Language Toolkit would have something useful for you. They have a great python api!

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.