I have a document corpus and I want to estimate the probability of occurrence of a certain word $w$. Simply calculating the frequencies and use such a number as an estimation is not a good choice. Is there any work on this topic describing a better approach?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ How big is your corpus? What language? Why do you consider "calculating the frequencies...is not a good choice"? Are you interested in common words, not-so-comment, very rare? I think a little more context would help. The broad subject matter is a fairly richly developed research area. :) $\endgroup$– cardinalCommented May 17, 2012 at 13:01
-
$\begingroup$ @cardinal: I think the frequency is not smooth enough and that zero frequency may occur but zero probability is not so good... $\endgroup$– derekhhCommented May 17, 2012 at 13:54
-
2$\begingroup$ So you're mostly interested in very rare words? What does a plot of the frequencies look like? Of a plot of rank vs. count? Updating your question with context of this sort would be very helpful. $\endgroup$– cardinalCommented May 17, 2012 at 14:04
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I think you want to have a look at what the text-mining people call smoothing. A simple smoothing technique is to add one to every word count, so no word has a zero probability estimate - essentially pretend that every word occurs once more than it does in reality. Generalized, this is sometimes called "Laplace smoothing" or "additive smoothing" - it's a form of shrinkage applied to the probability estimation.
Most of the time for simple problems add-one smoothing smoothing will work ok, so it's a good starting point if you're trying to get started, which is what it sounds like.
However there's many more techniques, and you need to be careful applying this "add one" to bigrams/n-grams. There's a very rich literature if you want to get into it. Look up Good-Turing Smoothing and Katz Smoothing, and m-estimate smoothing to get an idea of the flavor of these techniques.
$\begingroup$
$\endgroup$
It is hard to answer your needs without more detail. In text analysis, word frequencies are replaced by tf*idf which stands for "term frequency times inverse document frequency". This is an empirical score that corrects for the occurrence of terms that are frequent in the corpus and thus do not discriminate documents. It is widely used to compare texts, in particular through the cosine similarity measure.
In practice, you compute the frequency of the term in the document (tf) and multiply it by the log of the inverse fraction of documents containing the term (idf).
The site of Python's NLTK (natural language toolkit) contains an implementation of it, along with other tools and a good deal of explanations.
That said, if what you really want is an estimator of the probability of occurrence of the word, I don't know if you can get better than the frequency. And if 0 count is an issue, you can use the the Bayesian estimator (k+1) / (n+1), where k and n are word count and text size respectively.
Edit: for a great read about IDF, take a look at S. Robertson's paper Understanding Inverse Document Frequency
answered May 17, 2012 at 21:36