I'm actually working on an information retrieval project, and I want to extract words that are of significance from a sentence. This is somewhat opposite to stopwords.
In a sentence like: He was a great astrologer and an orthodontist.
... I want to extract out words that might be of more importance. To do this, I would definitely need to filter out the stop words, but words like 'orthodontist' and 'astrologer' are far more important since they do not occur frequently. I want to be able to extract such words for my training data.
Also, my need would be satisfied if there is some API that gives you the frequency of all words that are seen in a huge enough corpus (maybe a few million sentences?) that can probably represent the distribution of words in communication in English language.
For ex: I'd input 'astrologer' and I'd get a frequency score of 0.3 (this is random, and normalized) as opposed to getting a frequency score of 0.9 for 'the'